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EXECUTIVE  SUMMARY 


The  U.S.  Army  Environmental  Center  (USAEC)  has  an  established  program  to  assess 
technologies  suitable  for  the  detection,  identification,  and  excavation  of  unexploded 
ordnance  (UXO).  This  report  presents  the  results  of  the  third  series  (Phase  III)  of  UXO 
Advanced  Technology  Demonstrations  (ATD)  completed  at  Jefferson  Proving  Ground 
(JPG)  in  Madison,  Indiana.  The  analysis  documents  the  performance  capabilities  of  15 
demonstrators  who  participated  in  the  Phase  El  ATD,  and  compares  their  overall 
performance  to  what  was  achieved  in  two  earlier  Phases.  Demonstrators  in  all  three 
Phases  were  required  to  either  search/detect/characterize  or  excavate  inert  ordnance  that 
was  deliberately  emplaced  for  the  ATD.  The  performance  data  define  the  capabilities  and 
limitations  of  UXO  technologies,  as  demonstrated  under  the  JPG  test  conditions  and 
evaluated  by  the  ATD  methodology.  This  data  will  be  useful  to  those  who  wish  to  better 
understand  th^  challenges  posed  by  UXO,  and  to  those  who  may  have  to  respond  to  those 
challenges. 

The  need... 

UXO  technology  deficiencies  came  to  the  forefront  of  our  nation’s  newspapers  with  the 
public’s  realization  that  the  base  realignment  and  closure  (BRAG)  process  would  not 
result  in  the  immediate  turnover  of  formerly  used.  Department  of  Defense  (DoD) 
properties.  A  legacy  of  bombs,  missiles,  and  rockets  decades  old,  and  even  cannonballs 
from  the  past  century  restricts  unlimited  public  use  or  access  to  these  lands.  In  addition, 
active  DoD  installations  considering  alternative  land  uses  must  face  unknown  hazards,  as 
record  keeping  of  past  ordnance  usage  was  nonexistent  or  incomplete.  Installation 
managers  need  to  know  the  capabilities  of  UXO  technologies.  There  is  an  enormous 
demand  to  characterize  properties  just  so  the  extent  of  the  UXO  hazard  can  be  defined.  In 
addition,  there  is  a  demand  for  lands  to  be  returned  to  the  public  domain  through  UXO 
remediation  efforts.  UXO  cleanup  efforts  are  estimated  to  cost  in  the  tens  of  billions  of 
dollars. 
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The  response... 


The  U.S.  Congress  established  the  UXO-ATD  program  to  focus  technology  on  reducing 
the  unfunded  liability  and  the  time  needed  to  characterize  and  remediate  property. 
Congress  recognized  the  need  for  more  cost-effective  and  safer  technologies.  The 
USAEC  manages  the  ATD  program  at  JPG  under  the  Congressional  mandate  to 
demonstrate  advanced  and  innovative  UXO  technologies.  A  meaningful  framework  for 
understanding  UXO  technology  performance  was  established  by  publishing  public  criteria 
and  metrics.  The  ATD  program  would  not  only  benefit  restoration  managers,  who  need 
to  know  more  than  just  how  to  spell  “ordnance”  correctly,  but  also  technology  developers 
who  would  have  quantifiable  goals  to  seek  against  published  performance. 

Phases  I  and  II... 

In  the  first  two  phases,  conducted  in  1994  and  1995  respectively,  ordnance  was  emplaced 
that  was  representative  of  different  UXO  conditions.  Two  sites,  16  and  32  hectares,  were 
established  for  ground-based  and  airborne  technology  demonstrations.  There  were  29 
demonstrations  in  Phase  I  and  17  demonstrations  in  Phase  n.  These  demonstrations 
showed  that  airborne  platforms  and  ground  penetrating  radar  (GPR)  sensors  did  not 
perform  well  under  the  test  conditions  at  JPG.  Demonstrators  who  used  a  combination  of 
sensors  (electromagnetic  induction  and  magnetometry)  had  the  best  performance.  The 
better  performers  in  Phase  II  detected  over  80  percent  of  the  ordnance,  hut  they  also 
reported  three  to  twenty  times  more  targets  (false  alarms)  than  actual  ordnance.  The 
inability  to  distinguish  ordnance  from  the  prevalent  farming  debris  at  the  site  was  noted, 
because  this  would  likely  be  a  major  cost  factor  in  remediating  UXO  properties. 
Excavation  demonstrations  of  remotely  operated  systems  were  also  demonstrated  at  the 
two  Phases.  Excavators  could  unearth  ordnance  at  only  a  fractional  rate  (<5%)  of  how 
fast  demonstrators  could  detect  it. 

Phase  III... 

In  Phase  III,  the  ordnance  layout  was  changed  from  the  earlier  Phases  to  represent 
geographically-defined  UXO  scenarios.  An  Aerial  Gunnery  Range  (1),  Artillery  and 
Mortar  Range  (2),  Grenade  and  Submunition  Range  (3),  and  Interrogation  and  Burial 
Area  (4)  were  established  on  the  16  hectare  site.  Demonstrators  were  allowed  to  select 
the  scenarios  that  best  represented  their  system’s  capabilities  for  detection,  localization 
and  or  characterization  of  the  UXO.  Remote  excavation  technologies  were  also  solicited. 
Fifteen  proposals  were  funded  at  a  maximum  of  $75K.  One  company,  Sanford  Cohen 
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and  Associates  (SC&A)  formed  a  teaming  arrangement  with  three  survey  demonstrators 
(ADI,  Geo-Centers  Inc.,  and  Geometries)  to  apply  SC&A’s  advanced  data  processing  to 
their  data.  Geophysical  Research  Institute  (GRI)  reported  their  magnetometer  (Mag), 
electromagnetic  induction  (EM),  and  combined  sensor  (Combined)  target  data  separately. 
ADI  used  a  Mag  in  (1)  and  (2)  and  EM  and  Mag  in  (3).  The  overall  detection 
performance  of  the  Phase  HI  demonstrations  is  summarized  in  Table  ES-1,  as  categorized 
by  sensor  technology. 

TABLE  ES-1 

DEMONSTRATOR  ORDNANCE  DETECTION  BY  SENSOR  TECHNOLOGY 
COMBINED  SCENARIOS  (1, 2,  AND/OR  3) 


Sensor  Type 

Demonstrator  (Scenario  #) 

Pd 

False  Alarm 
(FA)  Rate 
(#/Hectare) 

FA  Ratio 
(#/Ordnance 
Detected) 

Electromagnetic 

Induction  (EM) 

CHEMRAD  (1,2) 

0.50 

12.90 

1.91 

GRI  (EM)  (1,2,3) 

0.87 

123.89 

8.46 

GeoPotential  (1,23) 

0.06 

9.04 

8.54 

Gradiometer  (Grad) 

Foerster  (1) 

0.60 

36.46 

4.85 

Magnetometer 

Battelle  (2) 

0.12 

1.71 

1.00 

(Mag) 

GRI  (Mag)  (1,2,3) 

0.70 

223.68 

18.82 

Rockwell  (1,2) 

0.34 

25.93 

5.70 

EM  &  Grad 

Geophex  (1,2) 

0.77 

32.44 

3.11 

ADI  (3;  Mag  only  in  1,2) 

0.78 

109.48 

8.30 

GRI  (Combined)  (1,2,3) 

0.93 

240.53 

15.23 

Geo-Centers  (1,2,3) 

0.93 

81.80 

5.18 

EM  &  Mag 

Geometries  (2) 

0.90 

38.44 

3.00 

NAEVA(1,2) 

0.94 

24.84 

1.96 

SCA_ADI  (3;  Mag  only  in  1,2) 

0.63 

46.80 

4.36 

SCA_Geo-Centers  (1,2,3) 

0.76 

43.55 

3.36 

SCA_Geometrics  (2) 

0.96 

41.86 

3.06 

Ground  Penetrating 

ENSCO  (1,2) 

0.70 

48.66 

5.14 

Radar  &  EM  &  Grad 

Note:  Detection  probabilities  are  based  on  detecting  all  the  ordnance  within  a  given  Scenario.  Battelle, 
CHEMRAD,  Foerster,  Geo-Centers,  and  GRI  did  not  survey  their  entire  Scenario(s). 
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The  table  shows  that  overall  performance  was  satisfactory,  as  many  demonstrators  found 
more  than  90  percent  of  the  baseline  ordnance.  The  comparison  of  these  results  to  the 
earlier  Phases  is  shown  in  figure  ES-1,  the  probability  of  ordnance  detection  versus  the 
false  alarm  rate  in  false  alarms  per  hectare.  Good  performance  is  in  the  upper-left  hand 
comer  of  the  plot.  The  general  trend  is  that  detection  is  improving  (movement  up  the 
plot)  but  target  discrimination  (false  alarm  rate)  has  not  changed  (no  movement  to  the  left 
edge  of  the  plot).  Localization  performance  for  ground-based  demonstrators  continues  to 
improve  since  Phase  I  as  shown  in  figure  ES-2.  Remote  target  excavation  feasibility  was 
shown,  but  target  excavation  can  take  one  half  hour  or  better  per  target. 

In  Summary,,. 

The  strengths  and  capabilities  of  UXO  technologies  were  demonstrated  to  show 
continued  and  satisfactory  improvement  in  detection  performance.  Because  there  has 
been  no  substantial  change  in  the  ability  of  demonstrators  to  discriminate  UXO  from  the 
clutter  at  JPG,  a  focused  effort  is  needed  to  resolve  this  issue.  A  poor  target 
discrimination  capability  means  remediation  efforts  will  likely  suffer  from  excessive 
expenditures  of  time  and  money.  A  strong  initiative  is  needed  to  encourage  the  further 
development  of  advanced  data  processing  and  new  approaches  that  can  address  this 
technology  deficiency.  It  is  recommended  that: 

•  Target  discrimination  goals  be  established. 

•  Standard  formats  for  raw  sensor  data  be  established. 

•  Factors  that  affect  ordnance  and  nonordnance  discrimination  be  identified. 

•  Raw  sensor  data  with  ground  truth  be  made  available  to  the  developers  of 
discrimination  algorithms. 

•  Innovative  and  high-risk  technologies  be  funded  for  further  development. 

•  Facilities  and  a  test  area  at  JPG  be  made  available  to  those  who  wish  to  use  it  for 
technology  development. 
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Figure  ES-1 


Rxjbability  of  Ordnance  Ddectico  (It)  vasus  False  AlamiRate  (EAR) 
Comparison  of  Phase  I,  Phase  n  and  Phase  IH 
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CONTROLLED  SITE 


ADVANCED  TECHNOLOGY  DEMONSTRATION  PROGRAM 

PHASE  III 
DRAFT  REPORT 

U.S.  ARMY  JEFFERSON  PROVING  GROUND 
MADISON,  INDIANA 


Prepared  for  U.S.  Army  Environmental  Center 


1.0  INTRODUCTION 


The  U.S.  Army  Environmental  Center  (USAEC)  has  an  established  program  to  assess 
technologies  suitable  for  the  detection,  identification,  and  excavation  of  unexploded 
ordnance  (UXO).  The  U.S.  Congress  initiated  this  program  to  identify  innovative 
technologies  that  would  provide  more  effective,  economical,  and  safer  methods  for 
removing  ordnance  from  lands  once  used  for  U.S.  Department  of  Defense  (DoD)  military 
training  and  testing  activities.  USAEC  has  established  a  project  to  meet  the  congressional 
mandate  by: 

•  Establishing  public  criteria  and  metrics  that  provide  a  meaningful  framework  for 
understanding  and  assessing  UXO  technology; 

•  Providing  an  opportunity  with  funding  for  demonstrators  to  undergo  an  unbiased 
assessment  of  their  UXO  technologies,  and  providing  these  demonstrators  target  data 
so  that  they  could  undertake  system  improvements; 

•  Documenting  the  performance  of  advanced  technologies  demonstrated  on  controlled 
sites  with  inert  ordnance  and  live  (ordnance)  sites  so  that  decision  makers  have  a 
better  understanding  of  the  capabilities  and  limitations; 

•  Seeking  to  improve  the  demonstration  methodology  so  that  the  results  are  more 
applicable  to  actual  UXO  clearance  operations  and  decision  making. 

This  report  covers  Phase  III,  the  third  in  a  series  of  controlled  site  demonstrations  that 
have  been  conducted  at  Jefferson  Proving  Ground  (JPG)  in  Madison,  Indiana.  The  Naval 
Explosive  Ordnance  Disposal  Technology  Division,  PRC  Environmental  Management 
Inc.  (EMI),  and  Litton/PRC  Inc.  provided  technical  assistance. 
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1.1  PROGRAM  BACKGROUND 


The  Base  Realignment  and  Closure  process  brought  increased  attention  to  the  problem  of 
UXO  on  Department  of  Defense  (DoD)  properties  and  Formerly  Used  Defense  Sites 
(FUDS).  The  methods  for  removing  UXO  from  the  land  were  of  questionable 
effectiveness,  labor-intensive,  and  costly.  HR  5504  established  the  Advanced 
Technology  Demonstration  (ATD)  program  to  identify  and  evaluate  technologies  for 
UXO  detection  and  remediation  (U.S.  House  of  Representatives  [USHR]  1992a,  1992b, 
1992c).  In  1994,  Phase  I  demonstrations  were  carried  out  in  at  JPG  on  two  sites  seeded 
with  representative  UXO  targets.  Meaningful  criteria  were  established  for  UXO 
detection,  localization,  and  classification.  In  1995,  the  Phase  11  controlled  site  program 
was  continued  at  JPG  while  additional  technology  demonstrations  were  being  conducted 
on  five  live  ranges  across  the  United  States.  The  conclusion  of  these  trials  and  the  limited 
funding  available  led  to  a  re-examination  of  the  ATD  program  objectives  for  Phase  EEI. 
Congress  (USHR  1994)  had  noted  in  their  funding  authorization  that  this  would  be  the 
last  year  that  they  would  direct  the  Army  to  continue  the  JPG  Project. 

The  JPG  ATD  philosophy  has  been  to  measure  the  system  performance  of  selected 
technology  demonstrators  and  not  examine  the  subsystem  (e.g.  sensor)  data  or  the 
intermediate  data  steps.  System  performance  evaluations  are  relevant  to  the  needs  of 
Government  decision  makers  and  Installation  managers.  With  this  resuits-oriented 
approach,  the  burden  is  placed  on  the  demonstrators  to  use  adequate  search  procedures 
and  to  make  those  tradeoffs  needed  to  accomplish  the  task  at  hand.  The  JPG  proposal 
selection  process,  demonstration  methodology,  and  evaluation  criteria  are  intended  to 
influence  the  choices  technology  users  make  to  solve  the  “UXO  problem”.  By  reviewing 
publicized  criteria  and  technological  capabilities,  developers  and  customers  of  these 
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technologies  have  a  better  sense  of  what  is  available  and  what  is  needed  in  the 
competitive  market  environment.  The  Phase  in  ATD  goals  are  designed  to  represent  the 
needs  of  the  UXO  community  as  guided  by  the  Congressional  charter.  In  order  to  put 
Phase  in  in  perspective,  it  is  first  necessary  to  review  the  objectives  of  the  earlier  phases 
and  some  of  the  issues  surrounding  the  ATDs. 

1.1.1  JPG  Phase  I. 

The  objectives  for  JPG  Phase  I  included  evaluating  existing  and  promising  UXO 
technologies.  Two  controlled  test  site  areas  were  prepared:  a  16-hectare  (40-acre)  site  for 
ground  system  demonstrations  and  a  32-hectare  (80-acre)  site  for  airborne  system 
demonstrations.  A  variety  of  inert  ordnance  and  nonordnance  items  were  emplaced  at 
depths  and  orientations  representative  of  formerly  used  defense  sites  and  active  military 
ranges  with  impact  sites  and  disposal  areas.  The  position  of  each  item  was  surveyed  to 
provide  a  means  to  measure  demonstrator  performance.  Technologies  were  solicited  that 
could  be  used  to  survey  either  site  within  five  days,  or  that  could  be  used  to  remotely 
excavate  ordnance. 

There  were  29  system  demonstrations  (some  companies  did  multiple  demonstrations) 
conducted  from  April  through  October  1994  (USAEC  1995).  These  demonstrations 
showcased  a  variety  of  sensor  platforms,  including  man-portable,  vehicle,  combinations 
of  man-portable  and  vehicle,  and  airborne,  and  also  included  three  remote  excavation 
demonstrations.  Sensor  technologies  included  magnetometer  systems,  electromagnetic 
induction  systems,  ground  penetrating  radar  (GPR)  systems,  infrared  systems,  and 
variants  or  combinations  of  the  above. 

The  results  showed  that  demonstrators  could  not  detect  more  than  two-thirds  of  all  the 
emplaced  ordnance  (less  than  40  percent  was  more  typical),  and  that  few  companies  (four 
demonstrations  of  twenty)  were  able  to  complete  a  search  goal  of  16  hectares  in  five  days. 
Airborne  systems  performed  poorly  per  the  detection  evaluation  protocol,  having  0  to  8 
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percent  detection  probability.  No  significant  target  classification  capabilities  were 
demonstrated  in  Phase  I,  and  multiple  false  targets  (typically  three  to  ten)  were  detected 
for  every  baseline  ordnance  item  detected.  The  excavation  systems  were  generally  slow 
(averaging  just  under  one  hour  to  excavate  a  target),  prone  to  breakdowns  (the  three 
systems  were  able  to  excavate  only  4  to  1 1  targets  in  40  hours),  but  at  least  proved 
successful  in  remotely  excavating  targets.  After  the  trials,  demonstrators  stressed  the 
need  for  pre-survey  site  visits  and  Phase  II  was  changed  to  accommodate  this 
recommendation. 


1.1.2  Phase  II 

The  objectives  of  Phase  II  included  the  continued  efforts  to  advance  the  UXO  technology 
performance  baseline  established  in  Phase  I,  and  to  identify  those  sensor  technologies  that 
provide  more  effective  clearance.  All  technology  developers  were  invited  to  submit 
proposals  to  participate  in  Phase  H;  however.  Phase  I  demonstrators  also  had  to 
document  significant  improvements  to  their  systems.  Proposals  for  ground-based 
technologies  that  did  not  support  a  survey  capability  of  16  hectares  in  5  days  were 
rejected  because  of  the  desire  to  link  the  proposal  cost  to  a  level  of  effort. 

A  total  of  17  demonstrations  were  conducted  from  May  through  September,  1995 
(USAEC  1996).  These  included  three  airborne  systems,  six  man-portable  systems,  two 
vehicle-towed  systems,  four  combined  man-portable  and  vehicle-towed  systems,  and  two 
excavation  systems.  The  15  detection  technologies  used  magnetometer,  electromagnetic 
induction,  and/or  ground  penetrating  radar  sensors.  Most  companies  were  able  to 
complete  the  16  hectare  survey  in  the  allotted  five  days,  while  the  airborne  systems 
completed  their  32  hectare  surveys  under  the  allotted  three  days.  Detection  probabilities 
improved  to  a  high  of  85  percent  with  most  ground-based  demonstrators  above  50 
percent. 
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The  results  showed  that  the  more  effective  demonstrators  used  electromagnetic  induction 
sensors  and/or  magnetometers.  Airborne  systems  that  used  sensors  similar  to  ground 
based  technologies  failed  to  show  any  significant  detection  capability  (less  than  5 
percent),  suggesting  that  the  platform  altitude,  speed  or  other  parameters  degraded 
performance.  Target  discrimination  continued  to  be  a  problem,  as  demonstrators 
typically  reported  between  4  and  20  times  more  targets  as  ordnance  than  the  number  of 
ordnance  items  they  had  actually  detected.  The  production  rate  of  excavation  systems 
improved  with  11  and  18  targets  remotely  excavated  by  two  systems  over  an  allotted  24 
hours. 


1.1.3  Live  Site  ATDs 

Five  ATDs  were  conducted  on  ordnance  ranges  in  Yuma  Proving  Ground  (Arizona), 
Eghn  Air  Force  Base  (Florida),  JPG  (Indiana),  McChord  Air  Force  Base  (Washington), 
and  Fort  Jackson,  South  Carolina  (UXO  Forum,  1996).  The  objective  was  to  ascertain 
the  performance  of  selected  JPG  Phase  I  demonstrators  surveying  working  on  160 
hectare  sites  with  different  environmental  and  debris  conditions  than  the  JPG  Controlled 
Site. 

The  ranges  were  surface  swept  of  hazards  and  surface  debris.  A  small  controlled  set  of 
baseline  inert  ordnance  was  emplaced  on  each  of  the  live  ranges  and  used  to  measure 
detection  performance.  Each  range  had  at  least  two  demonstrations  of  different 
technologies,  and  no  demonstrator  participated  in  more  than  two  ATDs.  The  detection 
results  for  the  demonstrators  were  comparable  to  those  achieved  in  the  Phase  I  controlled 
site  trials.  Demonstrators  who  performed  well  in  JPG  Phase  I  also  performed  well  at  the 
live  sites.  Demonstrators  who  performed  poorly  in  Phase  I  likewise  performed  poorly  on 
the  live  sites.  Thus  geophysical  conditions  did  not  significantly  affect  demonstrator 
search  performance.  Excavation  demonstration  performance  was  degraded  at  the  live 
sites.  Selected-target  validation  (excavation)  proceeded  at  the  rate  of  2  to  6  holes  a  day 
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on  “live  targets”.  This  was  a  serious  concern,  because  survey  demonstrators  reported  tens 
of  thousands  of  targets  on  some  sites. 

1.1.4  ATD  Controlled  Site  Issues/Limitations  to  Scope 

The  relevance  of  JPG  performance  data  to  other  sites  has  to  be  regarded  in  light  of  the 
demonstration  objectives  and  methodology,  and  the  dependency  on  the  local 
environmental  conditions.  One  objective  of  conducting  ATDs  at  a  fixed  site  such  as  JPG 
is  to  highlight  differences  in  performance  that  are  dependent  on  demonstrators’ 
technologies.  For  example,  performance  parameters  such  as  a  demonstrator’s  false 
alarms  (detected  nonordnance)  can  be  monitored  over  time,  as  these  tend  to  be  influenced 
by  the  sensor  technology  and  local  debris  conditions. 

ATD  resource  limitations  restrict  the  scope  of  testing  to  small  acreage,  so  that 
technologies  may  not  be  distinguished  based  on  economies  of  scale.  That  is,  technologies 
that  can  economically  survey  large  acres  will  not  be  highlighted.  In  addition,  other 
artificial  conditions  were  imposed  upon  the  demonstrators  so  that  the  “playing  field  was 
level.”  Both  JPG  Phase  I  and  II  afforded  demonstrators  advantages  and  constraints  that 
could  otherwise  affect  performance.  For  example,  demonstrators  were  not  allowed  to 
disturb  the  surface  as  they  might  do  to  characterize  noise  sources  or  to  remove  magnetic 
debris,  lest  they  affect  the  performance  of  demonstrators  that  followed.  Several  geodetic 
monuments  were  established  within  the  controlled  site  so  that  a  geodetic  reference 
(WGS-84  datum)  was  available  to  multiple  demonstrators  on  the  site  at  one  time.  The 
monuments  were  used  by  demonstrators  as  sites  for  differential  GPS  base  stations  and 
also  to  geolocate  relative  navigation  systems.  A  local  30.5  meter  ( 100  ft.)  square  grid  was 
laid  out  within  the  controlled  site  to  facilitate  ordnance  emplacement  and  to  control 
multiple  demonstrators  on  the  site  at  one  time.  Many  demonstrators  used  the  convenient 
grid  to  manage  and  control  their  search  activities.  Demonstrators  were  provided  the 
location  and  depth  of  four  known  ordnance  targets  in  an  adjacent  demonstrator  reference 
area,  so  that  they  could  make  equipment  checks  at  their  convenience.  The  controlled  site 


1-6 


was  surface  swept  by  Explosive  Ordnance  Disposal  personnel  so  the  demonstrators  could 
also  operate  at  JPG  with  a  reduced  level  of  concern  for  UXO  risk  (unintended 
detonation). 

Other  site  conditions  likely  affected  performance.  The  metallic  debris  on  the  controlled 
site  was  typical  of  past  farming  activity  and  not  the  abundant  shrapnel  usually  found  on 
live  sites.  The  heavy  clay  soil  adversely  affected  ground  penetrating  radar  technologies 
and  the  vegetation  likely  affected  infrared  technology.  Finally,  ATD  results  may  improve 
just  from  learning  curve  experience  -  repeat  demonstrators  can  be  expected  to  be  better 
prepared  logistically,  as  well  as  new  demonstrators  who  have  learned  from  the  problems 
of  others. 

Significant  progress  has  been  made  in  understanding  the  capabilities  and  limitations  of 
UXO  technologies.  The  Phase  I  ATD  allowed  meaningful  performance  comparisons  to 
be  made  for  different  demonstrators  attempting  similar  tasks.  The  Phase  n  ATD  was  also 
able  to  mark  technological  progress  against  the  Phase  I  benchmark. 

While  these  two  trials  established  that  no  “magic  wand”  was  demonstrated  that  could 
make  the  JPG  soil  “transparent”,  other  issues  remain.  First,  the  mandate  of  the  program, 
“Advanced  Technology  Demonstration”  allowed  only  those  innovative  or  proven 
technologies  that  had  a  reasonable  chance  to  complete  the  required  level  of  search  effort. 
The  search  effort  requirements  in  the  proposal  selection  process  may  have  eliminated 
unique  technologies  that  have  other  merits.  Since  target  characterization  capabilities  have 
proven  poor,  technologies  that  do  not  support  a  search  methodology  should  be  considered 
for  this  application.  Another  issue  is  that  the  JPG  Phases  I  and  II  challenged 
demonstrators  to  search  all  depths  for  the  entire  baseline  ordnance  target  set  that  ranged 
from  mines  and  small  mortar  rounds  to  large  bombs.  This  broad  level  of  UXO 
concentration  is  not  usually  collocated  in  a  single  site.  The  performance  of  survey 
technologies,  operated  under  time  constraints,  may  have  been  affected  by  opposing 
strategies  to  localize  both  small  and  large  ordnance.  The  U.S.  Army  Engineering  and 
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Support  Center,  Huntsville,  provided  suggestions  to  make  the  ATD  more  realistic. 
USAEC  incorporated  their  comments  and  modified  Phase  HE  to  address  many  of  the 
above  concerns. 

1.2  PHASE  m  PROGRAM  GOAL  AND  OBJECTIVES 

The  primary  goal  for  Phase  IE  is  to  provide  relevant  information  to  Government  decision 
makers  on  the  performance  of  UXO  technologies  demonstrated  in  more  realistic 
situations.  Customers  of  these  technologies  will  be  able  to  use  this  information  to  make 
choices  of  appropriate  UXO  technologies  based  on  demonstrated  capabilities  and 
limitations.  In  addition,  the  Phase  III  results  may  be  useful  for  establishing  clearance 
requirements.  Another  objective  for  Phase  HI  is  to  continue  the  characterization  of  UXO 
technologies.  Developers  and  users  of  UXO  technologies  will  have  a  better 
understanding  of  the  progress  that  has  been  made  since  the  inception  of  the  ATD,  and  the 
gaps  that  remain.  Government  resource  managers  may  also  be  able  to  use  the  Phase  DI 
results  to  establish  priorities  for  research  and  development  investments. 
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2.0  ADVANCED  TECHNOLOGY  DEMONSTRATION  METHODOLOGY 


The  general  approach  to  the  ATD  Phase  IE  was  similar  to  that  developed  in  the  earlier  two 
phases.  Controlled  sites  with  deliberately  implanted  ordnance  and  nonordnance  were  established 
at  E*G.  Survey  demonstrators  were  invited  to  propose  how  they  would  collect  and  process 
sensor  data  on  the  buried  ordnance.  Proposals  that  offered  “best  value”  were  accepted  and 
scheduled  at  the  site.  Excavation  proposals  were  accepted  that  offered  site-acceptable  (e.g.  non¬ 
explosive)  techniques  for  remotely  excavating  ordnance.  After  the  demonstrations  were 
conducted,  the  detection,  localization,  classification  and  excavation  results  were  evaluated  based 
on  the  methodology  implemented  in  Phase  E.  Phase  El  was  modified  from  earlier  phases  to 
make  the  ATD  “more  realistic”.  Specific  details  of  this  approach  follow. 

2.1  SITE  LOCATION  AND  DESCRIPTION 

% 

JPG  is  located  about  5  miles  north  of  Madison,  Indiana,  in  Jefferson,  Ripley,  and  Jennings 
counties.  The  facility  covers  about  22,365  hectares  (55,265  acres)  and  includes  former  firing 
lines  and  impact  areas.  The  base  was  used  for  over  50  years  to  test  ordnance  and  related  systems 
up  to  1995.  The  Indiana  Air  National  Guard  still  uses  the  facility.  The  Phase  IE  demonstrations 
were  conducted  on  a  16  hectare  area  in  the  northwest  quarter  of  Section  36,  Township  6  North, 
Range  10  East,  the  same  area  used  for  Phase  I  and  Phase  E  demonstrations  of  ground  systems. 
The  site  is  located  adjacent  to  access  roads  on  the  east  side  of  the  facility.  Detailed  information 
on  the  site  and  its  preparation  are  contained  in  Appendix  A. 
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2.2  TECHNICAL  APPROACH 


There  was  a  desire  to  allow  demonstrators  the  opportunity  to  operate  as  they  would  normally  do 
so  in  the  field,  and  with  as  few  restrictions  imposed  as  practical.  The  demonstrator  reference 
area,  geodetic  monuments,  and  the  30.5  meter  grid  established  for  the  earlier  phases  (see  Section 
1.1.3)  were  retained  for  Phase  IE.  The  time  constraints  were  relaxed  from  earlier  phases,  in  that 
demonstrators  were  allowed  to  propose  how  much  time  they  would  require  within  limits.  Survey 
demonstrators  were  also  allowed  to  select  an  intrusive  or  non-intrusive  aspect  to  their  proposed 
effort.  “Intrusive”  demonstrators  would  be  allowed  to  excavate  to  a  depth  of  15  cm  to 
characterize  noise  sources,  remove  shrapnel,  etc.  Intrusive  demonstrations  were  required  to  have 
an  acceptable  site  safety  plan  containing  detailed  procedures  for  soil  removal  and  identifying 
qualified  personnel  for  (potential)  UXO  removal.  Since  no  proposals  to  conduct  an  intrusive 
demonstration  were  submitted,  further  discussion  of  this  aspect  of  the  ATD  is  limited  in  the 
report.  Sites  for  non-intrusive  demonstrations  were  established  in  a  16  hectare  controlled  area. 
Non-intrusive  demonstrations  did  not  allow  demonstrators  to  disturb  the  site. 

There  was  also  the  desire  to  establish  scenarios  and  survey  requirements  that  were  representative 
of  realistic  UXO  problems.  Specific  scenarios  were  established  for  Phase  IE  that  bounded  the 
range  of  UXO  sizes  and  the  required  search  depths  for  the  demonstrators.  The  scenarios  had 
representative  UXO  that  would  be  found  on  an  aerial  gunnery  range,  artillery  and  mortar  range, 
and  a  grenade  and  submunitions  range.  Another  scenario,  an  interrogation/burial  area,  contained 
a  wider  variety  of  ordnance.  The  first  three  scenarios  were  reserved  for  demonstrators  who  could 
search,  localize  and  classify  targets.  The  fourth  scenario  was  set  up  to  determine  the  capability  of 
systems  that  could  classify  marked  targets.  Excavation  demonstrators  would  be  assigned  targets 
at  all  four  scenarios.  Demonstrators  were  allowed  to  select  the  scenario(s)  that  matched  their 
capabilities  and  were  allowed  to  propose  up  to  a  maximum  of  40  hours  on  site.  Demonstrators 
were  not  required  to  propose  how  they  would  allocate  their  hours  among  two  or  more  scenarios. 
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The  funding  of  proposals  was  limited  to  $75K,  and  proposed  efforts  that  exceeded  this  figure 
would  not  be  considered  within  the  scope  of  the  ATD.  Since  the  average  cost  in  Phase  n  was 
S102K  for  ground  based  survey  systems  and  $176K  for  airborne  systems,  the  $75K  funding 
ceiling  was  expected  to  limit  demonstrators’  proposed  efforts  and  possibly  restrict  responses.  A 
“best  value”  proposal  acceptance  criteria  was  used  to  obligate  demonstrators  to  consider  cost 
sharing.  That  is,  ail  things  being  equal,  a  proposal  encompassing  two  scenarios  would  be  rated 
more  highly  than  a  single  scenario  proposal. 

2.3  SITE  LAYOUT 

The  four  UXO  scenarios  were  established  to  represent  conditions  at  U.S.  military  installations 
that  are  candidates  for  restoration  or  alternative  use.  The  Phase  EH  Area  Layout  Plan  (PRC  EMI, 
1996a)  defined  the  target  composition  for  each  scenario.  The  specific  target  positions  of  the  Area 
Layout  Plan  are  not  presented  in  this  report  to  protect  the  integrity  of  future  demonstrations.  The 
Phase  m  layout  was  completed  approximately  one  month  prior  to  the  start  of  demonstrations  on 
3  September  1996.  The  Phase  m  layout  resulted  in  major  design  changes  to  the  emplaced  Phase 
n  baseline  target  layout: 

•  Ordnance  targets  were  geographically  segregated  to  simulate  more  uniform  contamination 
within  identifiable  areas. 

•  The  separation  distance  between  ordnance  items  increased  to  reduce  the  clustering  of 
ordnance:  the  percentage  of  targets  with  a  separation  distance  of  less  than  2  meters  decreased 
from  38  to  2  percent  and  the  median  separation  distance  increased  from  5  to  7  meters,  (see 

figure  2.3-1) 

•  The  average  depth  of  the  baseline  target  set  decreased  from  0.9  meters  in  Phase  11  to  0.4 
meters  in  Phase  II. 

•  Targets  previously  identified  as  “cluster”  (many  small  items  within  a  confined  area),  were 
excavated  and  removed  from  the  baseline. 
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As  a  result  of  target  relocation  efforts  and  scenario  construction,  1  nonordnance  and  12  ordnance 
Phase  n  target  positions  were  defined  as  “no  contest”  areas  within  the  areas  designated  for  use  by 
Phase  in  search  systems.  “No  contest”  means  no  credit  or  penalties  are  assigned  for 
demonstrator  target  detection  reports  associated  with  these  positions.  The  “no  contest” 
designation  for  the  one  nonordnance  target  resulted  from  the  failure  to  find  and  remove  brass 
shell  casings.  The  12  ordnance  positions  were  designated  “no  contest”  accordingly: 

•  (3)  were  ordnance  items  deeper  than  described  for  the  scenario.  These  targets  were  too  deep 
to  economically  excavate  and  were  intentionally  left  in  situ; 

•  (9)  were  associated  with  unsuccessful  removal  attempts  during  scenario  construction  and 
resulted  in  the  government  no  longer  having  confidence  in  the  target  positions. 

The  9  targets  that  had  been  the  object  of  unsuccessful  scenario  excavation  attempts  included: 

•  (2)  partially  removed  30mm  clusters  whereby  4  of  8,  and  7  of  8  of  the  individual  ordnance 
items  were  recovered; 

•  (1)  8  inch  projectile  that  had  been  a  Phase  n  excavation  target.  The  projectile  was  exposed 
and  identified  in  the  Phase  n  excavation  attempt,  and  then  covered  up; 

•  (1)  60mm  mortar  round  that  the  excavators  had  dug  to  0.5  feet,  not  realizing  that  the  target 
was  at  2.56  feet. 

•  (3)  targets  that  had  been  detected  in  1995  by  Phase  II  demonstrators;  a  90mm  projectile,  a 
152mm  projectile,  and  500  pound  bomb.  The  500  pound  bomb  was  excavated  to  a  planned 
depth  of  10.6  feet,  and  a  subsequent  record  search  showed  it  was  actually  at  6.56  feet  -  the 
target  disposition  is  unknown.  A  Phase  HI  excavator  also  failed  to  locate  the  90mm 
projectile; 

•  (2)  30mm  ordnance  clusters  that  had  not  been  detected  by  any  Phase  II  demonstrators; 

In  addition  to  the  above,  two  excavations  on  the  Interrogation/Burial  site  resulted  in  only  partial 
recovery  of  multiple  targets,  but  these  two  targets  have  no  consequence  on  demonstrator 
performance  evaluations  in  that  scenario. 
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(UXO  excavation  is  not  a  trivial  task  as  evidenced  by  above  problems  in  finding  and  removing 
known  targets  at  known  positions.  There  were  168  attempted  excavations  of  Phase  11  targets  for 
the  Phase  HI  site  preparation.  These  excavations  were  hindered  at  times  by  wet  conditions  and 
further  constrained  by  the  nominal  one  hour  maximum  allotted  to  complete  each  excavation 
task). 

The  guidance  provided  to  the  survey  and  excavation  demonstrators  and  the  descriptions  of  the 
scenarios  are  described  in  the  following  respective  sections. 

2.3.1  Scenario  1,  Aerial  Gunnery  Range 

The  Aerial  Gunnery  Range  scenario  was  described  to  demonstrators  as  a  4  hectare  site  with 
helicopter  and  fixed  wing  aircraft  delivered  ordnance.  Ordnance  could  range  in  size  from  2.75 
inch  rockets  to  2000  pound  bombs.  Demonstrators  were  required  to  search  the  area  for  ordnance 
to  a  depth  of  3  meters.  There  were  43  ordnance  targets  and  78  nonordnance  items  emplaced  at 
the  site.  There  are  7  “no  contest”  locations  within  this  scenario. 

2.3.2  Scenario  2,  Artillery  and  Mortar  Range 


The  Artillery  and  Mortar  Range  scenario  was  described  to  demonstrators  as  a  4  hectare  site  with 
assorted  types  of  ground  ordnance  that  would  be  fired  at  fixed  targets.  Ordnance  could  range  in 
size  from  60mm  mortars  to  8  inch  projectiles.  Demonstrators  were  required  to  search  the  area  to 
a  depth  of  1 .2  meters.  There  were  67  ordnance  targets  and  50  nonordnance  items  emplaced  at  the 
site.  There  are  3  “no  contest”  locations  within  this  scenario. 

2.3.3  Scenario  3,  Grenade  and  Submunition  Range 

The  Grenade  and  Submunitions  Range  scenario  was  described  to  demonstrators  as  4  hectares  of  a 
former  impact  area  that  was  converted  into  a  range  for  small  sensitively  fuzed  ordnance. 
Consequently  the  existence  of  larger  ordnance  could  not  be  ruled  out.  Demonstrators  were 
required  to  search  the  area  for  all  ordnance  to  a  depth  of  0.5  meters.  There  were  98  ordnance 
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targets  and  39  nonordnance  items  emplaced  at  the  site.  There  are  2  no  contest  locations  within 
the  scenario. 

2.3.4  Scenario  4,  Interrogation/Burial  Area 

The  Interrogation/Burial  Area  scenario  was  described  to  demonstrators  as  a  4  hectare  site  with 
assorted  aerial  and  ground  delivered  ordnance,  bum  and  burial  pits,  and  nonordnance  debris.  The 
air  delivered  ordnance  could  range  in  size  from  2.75  inch  rockets  to  2000  pound  bombs,  and  the 
ground  delivered  ordnance  could  range  in  size  from  60mm  to  8  inch  projectiles.  The  site  had 
been  surveyed,  and  target  reports  needed  localization  and  classification.  Demonstrators  were 
given  20  surface-marked  targets  and  their  geodetic  positions  for  characterization  and  told  that  the 
depth  of  the  targets  of  interest  did  not  exceed  2  meters. 

There  were  53  ordnance  targets  and  72  nonordnance  items  emplaced  at  the  site.  Three  sets  of 
baseline  targets  were  surface  marked  by  color  coding  for  characterization.  All  three  sets  had  the 
identical  number  of  ordnance  and  nonordnance  targets.  The  ordnance  size  and  class  distribution 
was  the  same  for  all  three  sets. 


2.3.5  Excavation  Demonstrations 

The  entire  16  hectare  site  was  available  to  demonstrators  who  could  demonstrate  remote 
excavation.  The  demonstrators  were  to  be  provided  ordnance  to  excavate  that  was  within  the 
stated  capabilities  of  their  systems.  Demonstrators  were  provided  baseline  target  positions  to 
excavate  as  many  targets  as  practical  within  their  proposed  hours.  After  10  baseline  targets  were 
attempted,  target  positions  of  “no  contest”  ordnance  targets  were  to  be  included. 

2.4  DEMONSTRATOR  SELECTION  PROCESS 

Candidates  for  Phase  III  demonstrations  were  sought  through  a  Commerce  Business  Daily 
solicitation  (CBD  1995).  Interested  parties  were  then  sent  information  packages  that  included 
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program  background.  Phase  I  and  II  summaries,  Phase  HI  goals  and  requirements,  and  the  criteria 
for  selection.  A  total  of  32  proposals  were  received  for  Phase  HI  consideration. 

Criteria  for  the  proposal  evaluation  included  the  following: 

•  Cost  in  terms  of  best  value  to  the  Government; 

•  Applicability  in  meeting  Phase  HI  (scenario)  objectives; 

•  Proposed  technology; 

•  Key  personnel,  alliances,  and  relevant  experienee. 

The  Government  selection  panel  selected  15  proposals  for  participation  in  Phase  IE.  Fourteen 
other  proposals  were  rated  acceptable,  but  had  to  be  rejected  because  of  limited  funding.  Three 
proposals  were  determined  to  be  outside  of  the  scope  of  the  ATD. 

\ 

2.5  DEMONSTRATION  PROCEDURES 

All  demonstrators  chosen  were  provided  with  a  demonstration  work  plan  (DWP)  (PRC  EMI 
1996b)  that  outlined  the  responsibilities  for  the  parties  involved  in  the  demonstrations.  The 
DWP  provided  site  background,  evaluation  criteria,  and  data  validation  information.  The  Safety, 
Health,  and  Emergency  Response  Plan  (SHERP)  was  included  in  the  DWP  and  served  as  a  guide 
for  day  to  day  activities. 

Demonstrations  were  generally  scheduled  to  start  on-site  by  Wednesday  or  Thursday  and 
conclude  by  the  following  Sunday  or  Monday.  The  day  preceding  the  start  of  demonstrations 
was  available  for  system  set  up  and  check  out  on  the  Demonstrator  Reference  Area,  a  small  area 
outside  the  controlled  site  that  had  four  ordnance  targets  at  known  depths.  Demonstrators  were 
provided  daily  weather  forecasts,  including  data  collected  from  an  on-site  weather  station  at  an 
0700  safety  briefing.  Detection  demonstrators  were  given  the  specified  amount  of  time  on  site 
to  collect  data  based  on  their  proposals.  They  could  budget  their  time  as  they  saw  fit  for  the 
different  scenarios.  Demonstrators  on  the  Interrogation/Buriai  Area  were  provided  20  surfaced 
marked  targets  for  characterization.  Excavation  technology  demonstrators  had  40  hours  on  site 
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to  demonstrate  their  system's  capability.  These  demonstrators  were  provided  with  baseline  target 
positions  to  excavate.  Demonstrators  were  not  permitted  to  remove  any  objects  from  the  site 
during  the  demonstration.  A  Site  Manager  recorded  demonstrator  day  to  day  activities  in  a  field 

log. 


2.6  DEMONSTRATION  DELIVERABLES 

Demonstrators  were  required  to  supply  three  categories  of  data  within  30  days  of  demonstration 
completion: 

•  Administrative  data  that  identified  the  company  and  roles  of  the  project  team. 

•  Equipment  data  that  identified  the  technologies  used  in  the  demonstration. 

•  Results  data  tliat  the  government  used  to  evaluate  demonstrator  performance. 

The  administrative  and  equipment  data  was  provided  to  the  government  in  a  Demonstrator 
Summary  Report.  A  data  entry  disk  was  provided  to  each  demonstrator  to  ensure  standard  data 
submission  to  the  government  for  evaluation. 

2.7  QUALITY  ASSURANCE 

The  quality  assurance  program  plan  (QAPP)  for  Phase  III  outlined  quality  assurance  (QA)  and 
quality  control  (QC)  procedures  for  the  UXO  ATD  program.  The  primary  focus  of  the  QAPP 
was  on  the  integrity  of  the  emplaced  baseline  target  set  (target  and  position)  and  the  transmission 
of  data  from  the  demonstrator  to  the  government.  A  validation  of  the  algorithms  used  to  assess 
system  performance  was  also  conducted. 


2-8 


Figure  2.3-1  Ordnance  Separation  Distance  Plot 


Phase  III  Controlled  Site:  Combined 


- -  Critical  Radius:  1,  2,  and  S  meters 
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3.0  EVALUATION  METHODOLOGY  FOR  JPG  III  DEMONSTRATORS 


The  data  analysis  for  Phase  III  performance  is  based  largely  on  the  methodology 
originally  developed  by  Automation  Research  Systems  Ltd.  (USAEC  1994)  and  the 
Institute  for  Defense  Analyses  (USAEC  1995),  and  applied  in  JPG  Phase  11  (USAEC 
1996).  Survey  and  excavation  demonstrators  are  separately  evaluated. 

The  evaluation  of  demonstrators  who  participated  in  the  three  Phase  HI  search  scenarios 
(Aerial  Gunnery  Range,  Artillery  and  Mortar  Range,  and  Grenade  and  Submunition 
Range)  is  based  on  their  ability  to  detect,  localize,  and  characterize  the 
ordnance/nonordnance  baseline  target  set  within  each  respective  scenario.  Demonstrators 
who  participated  in  the  Interrogadon/Burial  Area  Scenario  are  evaluated  on  their  ability  to 
localize  and  (characterize  targets  that  had  been  surface-marked  for  them.  Excavation 
demonstrators  are  evaluated  on  their  ability  to  acquire  and  excavate  marked  and 
unmarked  targets. 


3.1  DETECTION 

The  ability  to  detect  subsurface  ordnance  is  critical  to  the  success  of  UXO  site 
characterization  and  remediation  efforts.  There  is  the  need  to  know  “where”  a  UXO 
problem  exists  and  a  more  precise  need  of  “where  to  put  the  shovel”.  Demonstrators  can 
meet  these  requirements  by  using  a  variety  of  sensors  to  sample  the  environment  for 
anomalous  changes  in  the  background  caused  by  the  presence  of  ordnance.  Their  sensor 
sampling  strategy  is  influenced  by  economics  and  determined  in  part  by  their  decisions 
regarding  the  sensor  technology,  numbers  of  sensors  used,  sensor  sampling  rate,  lane 
spacing,  search  speed,  and  quality  assurance  with  respect  to  a  desired  search  objective. 
The  evaluation  of  detection  capability  at  JPG  is  based  on  the  results  of  those  decisions 
made  to  accomplish  a  predetermined  level  of  search  effort,  constrained  by  time  and 
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funding.  The  evaluation  process  does  not  include  the  demonstrator’s  hidden  decision 
making  processes,  nor  an  evaluation  of  their  sensor  data. 

3.1.1  Basis  for  Detection 

Demonstrators  report  their  search  results  in  a  tabular  format  of  target  positions  with 
associated  characteristics.  The  basis  for  declaring  that  a  demonstrator  has  made  an 
ordnance  detection  is  dependent  on  the  decision  that  the  demonstrator’s  target  report 
“matches”  an  emplaced  baseline  ordnance  target.  The  principal  criteria  is  the  horizontal 
separation  distance  between  a  baseline  target  (center  of  volume)  and  the  location  in  a 
demonstrator  target  report.  The  requirement  to  accurately  estimate  the  location  of  an 
emplaced  target  places  a  burden  on  a  demonstrator’s  sensor/navigation  integration.  For 
example,  demonstrators  who  would  normally  “mag  and  flag”  (manually  interpret 
magnetic/electromagnetic  anomalies  and  surface  mark  them  with  flags  for  later 
excavation)  are  required  to  provide  geodetic  target  positions  of  their  marks  in  the 
Universal  Transverse  Mercator  (UTM)  grid  reference.  One  consequence  of  this 
evaluation  methodology  is  that  any  capable  sensor  is  only  as  good  as  the  navigation 
technique. 

3.1.2  Criteria  for  Detection 

A  detection  is  credited  when  the  separation  between  a  target  report  and  baseline  target 
position  is  less  than  or  equal  to  a  critical  distance  (critical  radius,  Rcrit).  Rcrit  should  be 
based  in  part  on  the  uncertainty  one  can  tolerate  in  relocating  targets.  Detection  is  not 
dependent  on  characterization,  so  an  ordnance  detection  is  credited  regardless  whether  the 
demonstrator  categorizes  (“types”)  the  target  report  as  an  ordnance  item  (tme  positive)  or 
nonordnance  item  (“mistyped”,  not  “false  negative”  which  has  been  corrupted  in  the 
vernacular  to  mean  an  excavation  that  yields  no  (“negative”)  target).  Likewise  a 
nonordnance  detection  is  credited  whether  the  baseline  nonordnance  target  is  typed  as 
ordnance  (false  positive)  or  nonordnance  (tme  negative).  The  criteria  established  in 
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earlier  phases,  and  retained  in  Phase  III,  require  ground-based  demonstrators  to  localize 
baseline  targets  to  within  2  meters  and  airborne  system  demonstrators  to  localize  targets 
to  within  5  meters.  (All  demonstrator  reports  are  analyzed  at  1,2,  and  5  meters  Rent,  but 
these  results  are  not  all  included  in  the  report  because  of  the  volume  of  information). 

An  automated  Target  Matching  Algorithm  (TMA)  is  used  to  do  the  comparison  of 
demonstrator  target  reports  to  the  baseline  target  set  in  a  consistent  manner.  A  variety  of 
TMAs  were  developed  in  earlier  phases  because  of  the  different  options  available  to 
associate  demonstrator  target  reports  with  adjacent  baseline  targets.  TMA  Group  is  used 
to  produce  demonstrator  detection  statistics  for  Phase  HI  as  it  had  been  used  for  JPG 
Phase  n.  TMA  Group  associates  baseline  ordnance  targets  within  a  given  distance  (group 
radius)  of  each  other  as  a  single  group  target.  Nonordnance  baseline  targets  can  also  be 
associated  wiA  an  ordnance  group.  (The  specifics  of  this  process  are  presented  in 
US  AEG,  1996).  The  group  radius  used  for  Phase  m  is  2  meters,  the  same  as  had  been 
used  for  Phase  H.  Targets  in  close  proximity  to  each  other  contribute  to  the  overall 
detection  signature  of  the  group,  and  may  increase  the  group  signature  over  that  of  the 
individual  targets.  A  case  is  made  for  using  this  algorithm  in  that  when  a  detected  target  is 
excavated,  the  excavation  is  usually  checked  for  additional  anomalies.  In  theory, 
detecting  one  target  is  as  good  as  finding  the  rest.  However,  there  is  the  concern  that 
collocated  targets  are  likely  easier  to  detect  than  single  targets.  Consequently,  the 
baseline  target  set  for  Phase  HI  was  spread  out  relative  to  that  of  Phase  H  to  focus  more 
on  single  target  detection  versus  group  target  detection.  Because  the  baseline  ordnance 
targets  of  Scenarios  1,  2,  and  3  were  separated  by  more  than  2  meters  in  Phase  IH,  TMA 
Group  does  not  cause  any  baseline  ordnance  to  “group”  at  a  2m  group  radius.  It  does 
group  one  nonordnance  item  with  an  ordnance  target  in  the  Aerial  Gunnery  Range, 
Scenario  2. 
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3.1.3  Detection  Performance 

Probabilities  of  detection  (Pd)  are  computed  for  each  demonstrator  from  the  reported 
targets  that  match  baseline  ordnance  and  nonordnance  targets  within  the  scenario 
searched.  These  detection  probabilities  are  defined  as  follows: 

Po.ord  =  Probability  of  detection  for  ordnance 

=  (#  ordnance  detected)/(#  baseline  ordnance  in  scenario  surveyed) 

PD.nonord  =  Probability  of  detection  for  nonordnance 

=  (#  nonordnance  detected)/(#  baseline  nonordnance  in  scenario 
surveyed) 

(In  Phase  H,  Pd  was  based  on  area  searched.  Phase  IQ  Pds  are  presented  in  charts  and 
tables,  based  on  the  entire  scenario(s).  Results  based  on  area  searched  are  noted  for  a  few 
demonstrators  in  the  text  of  Section  6).  The  methodology  of  determining  detection  opens 
up  the  possibility  for  a  demonstrator  to  achieve  a  high  Pd  solely  by  generating  a  large 
number  of  target  reports.  Likewise,  demonstrators  with  poor  technique  and  a  low  Pd,  can 
also  expect  some  success  just  from  the  random  chance  that  a  target  report  is  coincident 
with  a  baseline  target.  A  detection  probability,  Prandom,  was  calculated  for  each  survey 
demonstrator  and  included  in  the  statistical  data  charts  as  a  reference  to  this  issue.  P random 
is  the  expected  fraction  of  baseline  ordnance  targets  that  would  be  detected  if  the  total 
number  of  target  reports  of  a  demonstrator  were  randomly  distributed  within  the  search 
area,  as  opposed  to  being  specified  by  the  demonstrator.  It  is  calculated  as  follows: 

P  random  =  1  ~  C 

where  X  =  tip 

number  of  demonstrator  reports 
probability  of  having  a  report  within  Rcrit 

K  ^Rcrit^ 

A 

Area  surveyed 


n  = 
P  = 

P  = 
A  = 
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A  Pramiom  that  is  closc  to  or  excecds  PD,ord  is  indicative  of  a  detection  capability  that  may 
be  due  to  chance.  Overlay  plots  of  demonstrator  reports  and  baseline  targets  are  useful 
indicators  of  randomness  or  navigation  error  and  bias,  but  are  not  provided  in  the  report 
to  protect  the  baseline  integrity. 

Another  issue  with  detection  performance  is  that  a  system  with  a  high  probability  of 
detection  may  be  of  little  practical  value  if  it  generates  an  excessive  number  of  target 
reports  that  do  not  correspond  to  ordnance.  Demonstrators  who  detect  and  report  the 
debris  from  historic  farming  activity  at  the  16  hectare  site  will  have  many  targets  that  do 
not  correspond  to  the  baseline  target  set.  Because  target  excavation/investigation  of 
nonordnance  targets  may  represent  significant  wasted  resources  in  site  remediation 
efforts,  various  related  “false  alarm”  measures  are  presented  in  the  results  charts.  “False 
alarms”  are  defined  as  demonstrator  target  reports  that  do  not  correspond  to  baseline 
ordnance  targets.  This  definition  includes  detected  farm  debris  declared  as  such  in 
demonstrator  reports  and  true  negatives,  detected  nonordnance  baseline  targets  that  were 
correctly  typed  nonordnance.  (It  may  be  argued  that  true  negatives  do  not  belong  in  “false 
alarms”,  but  generally,  the  number  of  true  negatives  is  small  compared  to  the  number  of 
target  reports  that  do  not  correspond  to  baseline  targets.  Also,  the  number  of  true 
negative  targets  is  often  comparable  to  the  number  of  “mistyped”  characterizations  - 
ordnance  detections  typed  as  “nonordnance”.  Mistyped  ordnance  characterizations  are 
not  excluded  from  ordnance  detection  statistics.  It  is  conceded  that  demonstrators  would 
have  better  false  alarm  metrics  by  not  reporting  their  nonordnance  declarations,  and  that 
demonstrators  used  different  nonordnance  reporting  strategies).  The  definition  of  “false 
alarm”  does  not  include  any  demonstrator  target  reports  that  fall  outside  the  defined 
boundaries  of  the  scenario,  nor  any  demonstrator  target  reports  matched  to  the  “no 
contest”  locations  discussed  in  Section  2.3.  Thus  there  are  differences  in  the  number  of 
targets  a  demonstrator  reports  for  a  scenario  and  the  numbers  that  are  presented  in  the 
results  summaries. 
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The  false  alarm  ratio  is  the  number  of  false  alarms  divided  by  the  number  of  detected 
ordnance  targets.  This  ratio  is  the  number  of  excavations  that  yield  no  ordnance  per 
productive  (yielding  ordnance)  excavation.  The  ratio  assumes  that  all  target  reports  get 
investigated  and  that  excavators  are  100  percent  efficient.  (There  may  be  a  hidden  cost 
with  a  “high”  false  alarm  ratio,  in  that  excavator  success  with  ordnance  may  decrease  if 
the  general  expectation  is  not  to  find  any  ordnance).  The  false  alarm  ratio  of  4  is 
arbitrarily  used  to  distinguish  demonstrators  in  the  discussion  of  results.  Demonstrators 
whose  false  alarm  ratio  is  less  than  4  would  have  ordnance  in  more  than  20  percent  of 
their  target  excavations.  The  probability  of  false  alarm,  PfA)  is  the  fractional  area  of  the 
surveyed  area.  A,  that  is  covered  by  the  number  of  false  alarms,  Nfa.  and  defined 
accordingly: 


Nfa  K  ^Rcrii^ 

'  - 

The  probability  of  false  alarm  represents  a  measure  used  to  determine  the  Receiver 
Operating  Characteristics  (ROC)  curve  of  a  detection  system,  a  standard  method  of 
comparing  detection  performance  of  different  systems.  ROC  curves  define  a  relationship 
of  Pd  and  Pfa  over  various  detection  threshold  values.  However,  they  require  more  data 
than  developed  in  this  report  to  have  any  statistical  significance,  and  are  not  presented. 
Detection  performance  results  for  scenarios  are  provided  in  the  demonstrator  statistical 
data  charts  of  Section  5.  Demonstrator  statistical  performance  summaries  are  provided  in 
Appendix  B. 


3.1.4  Performance  Assessment  Plots 

Demonstrator  detection  ability  as  a  function  of  target  depth  and  size  are  represented  in 
log-log  scatter  plots  for  each  demonstrator.  Figure  3.1.4-1  shows  a  sample  plot.  The  top 
of  the  plot  shows  the  near  surface  targets,  the  left  side  of  the  plot  contains  the  smaller 
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targets.  Targets  that  were  detected  are  represented  by  filled  in  squares,  while  missed 
targets  are  left  blank.  Performance  assessment  plots  can  quickly  highlight  the  capabilities 
or  limitations  of  demonstrators  in  detecting  small  or  deep  ordnance.  Scenario  specific 
performance  assessment  plots  are  provided  in  Section  5.  Demonstrator  summary  plots  of 
combined  survey  data  are  presented  in  Appendix  B  with  each  demonstrator. 

Demonstrator  detection  ability  should  not  be  considered  without  the  associated  false 
alarm  metrics.  While  readers  may  wish  to  focus  on  this  one  statistic  to  compare 
performance,  they  are  cautioned  against  doing  so  because  of  the  methodology  that  is  used 
to  define  detection.  Plots  of  Pd  versus  the  false  alarm  rate  and  Pd  versus  the  false  alarm 
ratio  are  presented  in  the  discussion  of  overall  results.  Section  6.2,  to  represent  the 
significance  of  their  relationship.  A  sample  plot  of  one  of  these  plots  is  shown  in  figure 
3.1.4-2.  In  this  plot,  better  performance  is  in  the  upper  left  hand  comer  -  a  high 
probability  of  detection  with  a  low  false  alarm  ratio.  This  region  defines  capable 
technologies  suitable  for  remediation.  The  left  side  of  the  plot  defines  low  false  alarm 
technologies  that  may  be  suitable  for  statistical  site  sampling,  even  with  poor  detection 
performance.  Systems  in  the  top  right  section  of  the  plot  may  require  an  auxiliary  target 
interrogation  system  to  reduce  false  alarms,  and  so  make  them  economically  feasible  for 
remediation  efforts.  Systems  in  the  bottom  right  section  of  the  plot  represent  detection 
technologies  that  are  unsuitable. 
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Figure  3 . 1 .4- 1  Sample  Performance  Assessment  Plot 


3.2  LOCALIZATION 


Localization  statistics  define  the  accuracy  demonstrators  achieved  in  determining  the 
position  oi  detected  ordnance  targets  in  three  dimensional  space,  (x,  y,  z)  or  (easting, 
northing,  and  depth).  Localization  statistics  were  computed  for  all  four  scenarios. 
However,  demonstrators  in  Scenario  4  were  provided  the  positions  of  the  targets  they 
interrogated;  consequently.  Scenario  4  is  not  used  in  any  combined  scenario  statistical 
data. 

Horizontal  target  location  determines  where  and  how  wide  a  hole  to  dig  to  ensure  that  a 
target  gets  unearthed.  Accurate  depth  positioning  can  speed  the  excavation  process,  as 
excavators  can  slow  down  as  they  near  an  ordnance  item  so  as  to  minimize  disturbance. 
The  target  depth  can  also  determine  whether  a  target  even  needs  excavation  for  surveys 
with  a  depth  limitation.  Positioning  inaccuracy  impacts  excavation  efficiency,  ordnance 
safety  issues,  and  Occupational  Safety  and  Health  Administration  considerations.  Thus, 
there  is  a  need  to  characterize  demonstrator  localization  performance. 

3.2.1  Basis  for  Localization 

The  demonstrator  target  report  data  used  to  determine  baseline  ordnance  target  detection 
is  also  used  to  measure  localization  performance.  Demonstrators  who  participated  in  the 
Interrogation/Burial  Area  Scenario  are  required  to  locate  targets  within  the  critical  radius 
for  evaluation.  Whereas  TMA  Group  is  used  for  detection  statistics,  it  is  not  useful  for 
determining  localization  statistics  because  of  the  need  to  determine  which  item  in  an 
ordnance  group  likely  corresponds  to  the  demonstrator  target  report.  TMA  Closest  was 
developed  in  earlier  phases,  with  the  simple  rule  that  a  demonstrator  target  report  is 
assigned  to  the  nearest  ordnance  target  in  horizontal  space,  and  is  used  in  Phase  in  for 
localization  and  characterization  statistics.  (Because  the  baseline  ordnance  target  set  was 
dispersed  in  Phase  IH,  TMA  Group  and  TMA  Closest  are  equivalent  for  all  practical 
purposes). 
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3.2.2  Localization  Performance 


Positioning  errors  for  each  demonstrator  were  computed  from  the  set  of  baseline 
ordnance  targets  that  were  detected  using  TMA  Closest.  As  a  result,  horizontal  location 
errors  are  constrained  by  the  Rcrit  (2  or  5  meters)  used  in  the  detection  algorithm.  Depth 
errors  are  also  somewhat  constrained  because  search  depths  were  limited  in  each  of  the 
four  scenarios.  The  error  of  the  demonstrator  report  relative  to  the  approximate  center  of 
volume  of  the  detected  baseline  ordnance  target  is  computed  as  follows: 

easting  error:  dx  =  Xr  -  Xb 

northing  error:  dy  =  yr  -  yb 

depth  error:  dz  =  Zr  -  Zb 

s 

where  the  subscript  “r”  refers  to  the  demonstrator  report  and  “b”  refers  to  the  baseline 
target.  Negative  values  in  dx,  dy  and  dz  refer  to  demonstrator  reports  that  are  west  and 
south  of,  and  shallower  than,  the  baseline  target.  In  addition,  the  horizontal  radial  error,  r, 
is  computed  for  each  target  report  accordingly: 

radial  error:  r  =  ^jdx^  +  dy^ 

The  mean  and  standard  deviations  are  provided  for  dx,  dy,  dz,  and  r  as  computed  by 
standard  mathematics.  In  addition,  the  root  mean  absolute  depth  error  is  provided: 


absolute  depth  error:  \d^  = 
where  N  is  the  number  of  baseline  ordnance  items  detected. 

Localization  performance  results  are  provided  in  the  demonstrator  statistical  data  charts 
of  Section  5. 
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3.3  CHARACTERIZATION 


Characterization  studies  define  the  ability  of  demonstrators  to  provide  additional 
information  on  their  detected  targets  and  they  also  provide  some  discrimination  on 
demonstrator’s  detection  capabilities.  Characterization  statistics  were  computed  for  all 
four  scenarios. 

Target  characterization  is  necessary  because  large  amounts  of  nonordnance  debris  that  do 
not  require  removal  are  usually  encountered  where  UXO  is  present.  The  ability  to 
discriminate  between  ordnance  and  nonordnance  items  (referred  to  as  “typing”  in  this 
analysis)  is  very  important  in  reducing  the  number  of  nonordnance  items  (false  alarms) 
that  are  excavated.  In  addition,  excavators  would  benefit  from  target  descriptions  so  they 
are  aware  of  the  potential  hazards,  employ  proper  safety  precautions,  and  have  a  better 
idea  of  what  to  look  for  in  the  hole  or  overburden.  Thus,  there  is  a  need  to  measure 
demonstrator  characterization  performance. 


3.3.1  Basis  for  Characterization 

Demonstrators  were  requested  to  characterize  their  target  reports,  but  not  all  did  so  or 
they  only  reported  ordnance  detections.  Demonstrator  targets  that  corresponded  to  the 
emplaced  baseline  target  set  per  TMA  Closest  were  used  to  assess  “typing”  performance. 
Typing  required  demonstrators  to  declare  a  target  report  as  ordnance  or  nonordnance. 

Detected  baseline  ordnance  targets  were  used  to  assess  sizing  and  classifying  capabilities. 
Targets  are  sized  by  their  principal  diameter  as  small  (<  100  mm),  medium  (between  100 
and  200  mm)  and  large  (>  200  mm).  Demonstrators  were  asked  to  estimate  the  mass  of 
their  targets,  but  this  data  was  not  evaluated  and  consequently  is  not  incorporated  into  the 
report. 
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Demonstrators  could  further  classify  targets  as  bomb,  projectile,  mortar,  submunition  or 
rocket.  (Note:  while  Scenario  4,  the  Interrogation/Burial  Area,  was  identified  as  having 
potential  UXO  burial  pits  and  bum  pits  in  the  baseline  ordnance  target  set,  demonstrators 
had  no  such  classification  option.  No  burial  pit  or  bum  pit  targets  were  provided  to 
demonstrators  to  interrogate). 


3.3.2  Characterization  Performance 


To  assess  demonstrator  characterization  performance,  the  following  statistics  were 
computed  from  detected  baseline  items: 


Ability  to  Type: 

Pco  =  Probability  of  correct  characterization  of  ordnance 
=  (#  ordnance  correctly  typed)/(#  ordnance  detected) 

PcNo  =  Probability  of  correct  characterization  of  nonordnance 

=  (#  nonordnance  correctly  typed)/(#  nonordnance  detected) 

Ability  to  Size: 

PcsM  =  Probability  of  correct  size  determination  of  small  ordnance 

=  (#  small  ordnance  correctly  sized)/(#  small  ordnance  detected) 

PcMED  =  Probability  of  correct  size  determination  of  medium  ordnance 

=  (#  medium  ordnance  correctly  sized)/(#  medium  ordnance  detected) 

PcLG  =  Probability  of  correct  size  determination  of  large  ordnance 

=  (#  large  ordnance  correctly  sized)/(#  large  ordnance  detected) 

Ability  to  Classify: 

PcB  =  Probability  of  correct  bomb  classification 

=  (#  bombs  correctly  classified)/(#  bombs  detected) 

PcpRO  =  Probability  of  correct  projectile  classification 

=  (#  projectiles  correctly  classified)/!#  projectiles  detected) 
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PcMOR  =  Probability  of  correct  mortar  classification 

=  (#  mortars  correctly  classified)/!#  mortars  detected) 

PcsuB  =  Probability  of  correct  submunition  classification 

=  (#  submunitions  correctly  classified)/!#  submunitions  detected) 

PcR  =  Probability  of  correct  rocket  classification 

=  !#  rockets  correctly  classified)/!#  rockets  detected) 

In  addition  to  the  above  characterization  ratios,  the  detection  ratios  are  also  computed  for 
the  respective  categories  using  TMA  Closest.  Characterization  performance  results  are 
provided  in  the  demonstrator  statistical  data  charts  of  Section  5.  Some  precautions  are 
necessary  in  viewing  these  statistics  and  assumptions  are  necessary  in  determining  a 
demonstrator’s  capability: 

No  distinction  is  provided  for  demonstrators  who  didn’t  try  to  characterize  targets 
versus  those  who  tried  and  failed.  !If  all  values  are  “0”  in  the  “#  eorrect”  for  type, 
size,  or  classify  characterizations,  then  the  demonstrator  probably  elected  not  to 
attempt  that  particular  characterization). 

It  is  possible  to  achieve  a  100  percent  correct  characterization  of  ordnance  targets 
as  ordnance,  true  positives,  just  by  declaring  all  targets  as  ordnance.  This  in  itself 
is  not  a  true  measure  of  ordnance  typing  capability.  If  a  demonstrator  only 
reported  targets  believed  to  be  ordnance  and  therefore  had  no  reason  to  declare 
any  targets  as  nonordnance,  then  their  baseline  nonordnance  detection  statistics 
need  to  be  examined  to  infer  their  typing  ability.  In  this  later  case,  given  that  the 
demonstrator  has  a  nominal  ordnance  detection  capability  !50  percent  or  better),  a 
probability  of  nonordnance  detection  less  than  one  half  of  their  ordnance  detection 
probability  is  assumed  to  indicate  an  ability  to  type.  For  example,  a  demonstrator 
who  only  reports  ordnance  targets  has  a  Pco  of  0.80;  it  is  presumed  that  the 
demonstrator  would  need  a  Pcno  of  less  than  0.40  to  have  demonstrated  a  typing 
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capability.  That  is  an  indication  that  the  demonstrator  is  rejecting  nonordnance 
targets  as  ordnance. 

A  demonstrator’s  measured  ability  to  size  and  classify  baseline  ordnance  may  be 
biased  because  two  of  the  scenarios  narrowed  the  range  of  expected  sizes  and 
classes  of  ordnance.  Finally,  the  characterization  ratios  are  presented  as  point 
estimates,  and  they  may  lack  significant  statistical  confidence  because  of  the  small 
sample  size.  Also,  the  value  of  a  demonstrator’s  classification  statistics  is 
questionable,  if  demonstrators  lack  the  basic  ability  to  distinguish  ordnance  from 
nonordnance. 

3.4  EXCAVATION 

The  ability  to  excavate  targets  is  essential  for  target  identification  and  UXO  remediation. 
Excavation  technology  is  needed  that  can  traverse  uneven  terrain,  operate  safely,  operate 
against  a  variety  of  targets  at  different  depths^,  minimize  human  intervention,  and  achieve 
high  production  rates.  Excavation  demonstrators  were  assigned  target  positions  to 
excavate  that  were  within  their  system’s  capabilities.  They  were  required  to  provide  the 
travel  time,  dig  time,  target  depth,  and  target  identity. 
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4.0  DEMONSTRATION  SUMMARY 


This  section  is  a  brief  introduction  to  the  demonstrations  that  took  place  at  JPG  in  Phase 
TTT.  Not  all  demonstrators  participated  in  all  scenarios,  nor  did  all  demonstrators 
completely  survey  each  assigned  scenario.  Section  5  presents  the  performance  data  for 
each  demonstrator  separated  by  scenario.  The  results  of  the  excavation  demonstrations  are 
presented  in  Section  6,  the  Results  Summary,  Analysis  and  Conclusions.  Combined 
statistical  data  charts  and  performance  assessment  plots  (Scenarios  1, 2,  and  3  only)  for 
each  demonstrator  are  provided  in  Appendix  B,  Demonstrator  Summary  Reports  and 
Results.  Appendix  B  also  contains  detailed  information  on  the  demonstrators  and  their 
technologies,  as  provided  by  the  demonstrators. 

4.1  NON-INTRUSrVE  SURVEY  DEMONSTRATIONS 

Twelve  demonstrators  conducted  non-intrusive  demonstrations  in  September  through 
November  1996,  in  one  or  more  of  the  three  survey  scenarios  (1, 2,  and  3).  One  of  these 
companies.  Geophysical  Research  Institute  (GRI),  submitted  three  separate  target  reports 
for  data  gathered  with  two  sensors:  magnetometer  (Mag),  electromagnetic  (EM),  and 
combined  sensors  (Combined).  GRI’s  results  were  treated  as  separate  demonstrations 
and  are  annotated  as  GRI  (Mag),  GRI  (EM)  and  GRI  (Combined).  A  thirteenth  company, 
Sanford  Cohen  and  Associates  (SC&A,  or  SCA)  made  agreements  with  three 
demonstrators,  ADI  Limited  (ADI),  Geo-Centers,  and  Blackhawk  Geometries 
(Geometries),  to  post-process  their  respective  raw  sensor  data  and  submit  it 
independently.  These  results  are  treated  as  separate  demonstrations  and  are  annotated 
respectively  as  SCA_ADI,  SCA_Geo-Centers,  and  SCA_Geometrics. 

Battelle  Pacific  Northwest  Laboratories  (Battelle),  CHEMRAD,  Foerster,  Geo-Centers, 
and  GRI  (EM)  partially  surveyed  their  respective  scenario(s).  Since  Foerster  had 
originally  proposed  to  survey  only  half  (2  hectares)  of  Scenario  1,  their  ordnance 
detection  performance  based  on  the  actual  area  searched  is  noted  in  Section  6.1.1. 
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Foerster  also  elected  to  survey  half  of  Scenario  2  while  at  JPG;  however,  these  results  are 
not  provided  in  this  report.  CHEMRAD  was  the  only  company  whose  detection 
performance  was  significantly  affected  by  the  scenario-based  evaluation  methodology, 
and  their  results  are  noted  in  the  respective  scenarios. 

Eight  of  the  thirteen  companies  that  participated  in  the  survey  portion  of  Phase  in  ATD 
had  prior,  on-site  experience.  SC&A  also  had  prior  experience  analyzing  JPG 
demonstrator  data.  Table  4.1-1  lists  the  survey  demonstrators,  sensor  and  navigation 
technologies  used,  proposed  scenarios,  hours  proposed,  funding,  and  past  JPG  Phase 
experience. 

4.2  INTRUSIVE  SURVEY  DEMONSTRATIONS 

No  proposals  for  intrusive  demonstrations  were  received.  Demonstrators  were  allowed  to 
propose  an  intrusive  demonstration  that  would  allow  them  to  dig  into  the  soil  to  a  depth 
of  15  cm  to  characterize  and/or  remove  debris.  Possible  reasons  for  the  lack  of  proposals 
include:  a  $75K  ceiling  on  funding  would  limit  the  money  that  could  be  allocated  to 
intrusive  efforts;  the  40  hour  demonstration  time  limit  may  have  been  too  restrictive  for 
realizing  benefits  from  intrusion;  and  the  scope  of  nonordnance  debris  at  the  JPG  site  is 
not  the  unknown  as  it  was  to  Phase  I  demonstrators. 


4.3  CHARACTERIZATION  DEMONSTRATIONS 

One  of  the  goals  of  the  ATD  was  to  seek  technologies  that  could  better  localize  and/or 
characterize  ordnance  without  necessarily  being  suitable  for  conducting  general  search. 
Scenario  4,  the  Interrogation/Burial  Area,  was  specifically  established  to  test  these 
technologies.  One  proposal  was  received  that  was  directed  only  to  this  scenario,  and  it 
was  not  funded.  There  were  5  demonstrations  of  systems  in  Scenario  4  that  had  also 
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participated  in  one  or  more  of  the  search  scenarios.  SC&A  evaluated  two  of  these  data 
sets  for  a  total  of  7  demonstrations. 

4.4  EXCAVATION  DEMONSTRATIONS 

The  ATD  restricts  excavation  technology  proposals  to  remote  techniques  and  to  what  is 
considered  suitable  for  the  site.  Two  demonstrations  of  remote  excavation  technology 
were  conducted  in  November  1997.  OAO  had  proposed  40  hours  of  effort  for  $28.9K  and 
Lockheed  Martin  had  proposed  24  hours  of  effort  for  $74.4K. 


N 
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TABLE  4.1-1  DEMONSTRATION  SUMMARY 
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5.0  RESULTS  -  SCENARIO  ASSESSMENTS 


This  section  presents  the  individual  demonstrator  results  data  for  the  four  scenarios.  A 
statistical  data  chart  and  performance  assessment  plot  are  provided  for  each  survey 
demonstration.  The  statistical  data  charts  are  provided  for  Scenario  4.  A  combined 
statistical  summary  of  the  survey  scenarios  (1,2,  and  3)  is  provided  for  each  demonstrator 
in  Appendix  B. 


5.1  AERIAL  GUNNERY  RANGE 

This  section  presents  the  performance  of  the  following  demonstrators  that  participated  in  this  scenario: 


5.1.1 

ADI 

5.1.2 

CHEMRAD 

5.1.3 

ENSCO 

5.1.4 

Foerster 

5.1.5 

Geo-Centers 

5.1.6 

Geophex 

5.1.7 

GeoPotential 

5.1.8 

GRI  (Combined) 

5.1.9 

GRI  (EM) 

5.1.10 

GRI  (Mag) 

5.1.11 

NAEVA 

5.1.12 

Rockwell 

5.1.13 

SCA_ADI 

5.1.14 

SCA_GeoCenters 

5.1-1 


ADI  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

43 

34 

Nonordnance 

77 

67 

0.87  ! 

Total 

120 

101 

Number  False  Alarms 

360 

False  Alarm  Rate  (#/Hectare) 

104.16 

Fsilse  Alarm  Ratio  (#/Ord.) 

10.59 

Probability  False  Alarms 

0.131 

Localization  Statistics  (TMA  Closest)  (in  meters) 


1 

Mean 

Std  Deviation 

iPosition  (X.V)  _ 11 

dx  -  easting  error 

-0.05 

0.37 

dy  -  northing  error 

0.24 

0.41 

Radial  error 

0.48 

0.36 

iDepth  (z) _ J 

dz  -  averaged  depth  error 

0.29 

0.6  I 

Idzl®-  absolute  depth  error 

0.66 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  |  #  Detected 

Pd  1#  Correct  |  Pc° 

f  \  INI  'll  ti'l  I 

Ordnance 

43 

34 

0.79 

31 

0.91 

Nonordnance® 

78 

67 

0.86 

2 

0.03 

Large 

11 

11 

1.00 

10 

0.91 

Medium 

7 

6 

0.86 

0 

0.00 

Small 

25 

17 

0.68 

1 

0.06 

Abilitv  to  Classify 

Bomb 

21 

21 

1.00 

13 

0.62 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

13 

0.59 

2 

0.15 

Notes:  *  Target  Matching  Algorithm 

"  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
"  Probabiiity  of  correct  characterization 
®TMA  affects  how  nonordnance  baseiine  targets  are  counted 
NA  -  not  applicable 


5.1. 1-1 


JPG  Controlled  Site:  Phase  ill 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  ADI 

Small  Medium 


0.1 


£ 

o 


£ 

O. 

9 

Q 


1.0 


10.0  L 
10 


I  I  I  I  1  1 _ L- - 1 - 

100 

Principal  Diameter  (millimeters) 


■  Target  Detected 
□  Target  Not  Detected 


5. 1.1-2 


CHEMRAD  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA^  Group) 


.1 - - 

#  Baseline 

#  Detected 

Pd“ 

Ordnance 

43 

26 

0.60 

0.03  1 

Nonordnance 

77 

27 

0.35 

Total 

120 

53 

Number  False  Alarms 

57 

False  Alarm  Rate  (#/Hectare) 

16.49 

False  Alarm  Ratio  (#/Ord.) 

2.19 

Probability  False  Alarms 

0.021 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  easting  error 

-0.17 

0.67 

dv  -  northing  error 

0.2 

0.9 

Radial  error 

1.02 

0.51 

iDepth  (z) _ _ _ _ _ 

dz  -  averaged  depth  error 

0.31 

0.32  1 

ldzl°-  absolute  depth  error 

0.45 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Pd 

#  Correct 

Ordnance 

43 

26 

0.60 

0 

0.00 

Nonordnance® 

78 

27 

0.35 

0 

0.00 

Abllitv  to  Size 

Large 

11 

4 

0.36 

0 

0.00 

Medium 

7 

6 

0.86 

0 

0.00 

Small 

25 

16 

0.64 

0 

0.00 

Abilitv  to  Classify  1 

Bomb 

21 

9 

0.43 

0 

0.00 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

17 

0.77 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

"  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
*'  Probability  of  correct  characterization 
^MA  affects  how  nonordnance  baseline  targets  are  counted 
NA  •  not  applicable 


5. 1.2-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  CHEMRAD 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 
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ENSCO  •  Aerial  Gunnery  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Po^ 

Ordnance 

43 

30 

0.70 

0.O79  1 

Nonordnance 

77 

54 

0.70  1 

Total 

120 

84 

Number  False  Alarms 

192 

False  Alarm  Rate  (#/Hectare) 

55.55 

False  Alarm  Ratio  (#/Ord.) 

6.40 

Probability  False  Alarms 

0.069 

Localization  Statistics  (TMA  Closest)  (in  meters) 


1  Mean  I  Std  Deviation  | 

. II II  — 

dx  -  eastinq  error 

-0.24 

0.33 

dy  •  northing  error 

0.13 

0.61 

Radial  error 

0.58 

0.46 

Depth  fz) _ _ _ _ _ 

dz  -  average^  depth  error 

-0.02  0.34 

ldzl°-  absolute  depth  error 

0.35 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Pd 

#  Correct 

Pc*'  1 

Ordnance 

43 

30 

0.70 

0 

0.00 

Nonordnance® 

78 

54 

0.69 

0 

0.00 

Large 

11 

11 

1.00 

0 

0.00 

Medium 

7 

4 

0.57 

0 

0.00 

Smail 

25 

15 

0.60 

0 

0.00 

.  Ill . 

Bomb 

21 

21 

1.00 

0 

0.00 

Proiectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

9 

0.41 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
‘  Square  root  of  the  mean  square  depth  error 
^  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5.1.3-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
E}emonstrator:  ENSCO 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5-1. 3-2 


Foerster  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA^  Grou 


Notes:  *  Target  Matching  Algorithm 

"  Probability  of  detection 
®  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
*TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  >  not  applicable 


5.1.4-1 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  Foerster 


Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5. 1.4-2 


Geo-Centers  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA^  Group) 


#  Baseline 

#  Detected 

Ordnance 

43 

43 

1.00  1  0.109 

Nonordnance 

77 

63 

0.82  1 

Total 

120 

106 

Number  False  Alarms 

271 

False  Alarm  Rate  (#/Hectare) 

78.41 

False  Alarm  Ratio  (#/Ord.) 

6.30 

Probability  False  Alarms 

0.099 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  eastinq  error 

-0.38 

0.3 

dv  -  northing  error 

0.09 

0.34 

Radial  error 

0.53 

0.27 

iDepth  (z) _ _ _ _ 

dz  -  average^  depth  error 

0.59 

0.32  1 

IdzP-  absolute  depth  error 

0.68  II 

Characterization  Statistics  (TMA  Closest) 


W1  ^ T - - 

#  Baseline  #  Detected  Pd  |#  Correct  |  Pc  I 

1 1  III!  'll 

Ordnance 

43 

43 

1.00 

43 

1.00 

Nonordnance® 

78 

63 

0.81 

0 

0.00 

Large 

11 

11 

1.00 

7 

0.64 

Medium 

7 

7 

1.00 

3 

0.43 

Small 

25 

25 

1.00 

4 

0.16 

lAbilitv  to  Classify 

Bomb 

21 

21 

1.00 

21 

1.00 

Proiectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

22 

1.00 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

^  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
*TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5.1.5-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  GEO-CENTERS 

Small  Medium  Large 


Principal  Diameter  (millimeters) 


■  Target  Detected 
□  Target  Not  Detected 
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Geophex  •  Aerial  Gunnery  Range 


Detection  Statistics  (TMA^  Group) 


- r  I - 

#  Baseline 

#  Detected 

Pd*’ 

Ordnance 

43 

40 

Nonordnance 

77 

59 

0.77  1 

r 

Total 

120 

99 

Number  False  Alarms 

165 

False  Alarm  Rate  (#/Hectare) 

47.74 

False  Alarm  Ratio  (#/Ord.) 

4.13 

Probabilitv  False  Alarms 

0.060 

Localization  Statistics  (TMA  Closest)  (in  meters) 

Mean  Std  Deviation 

Position  (x.v) 

dx  -  eastinq  error 

-0.12 

0.45 

dv  -  northing  error 

0.13 

0.4 

Radial  error 

0.52 

0.35 

Depth  (z) 

dz  -  averaged  depth  error 

^^013  1 

0.26 

ldzl“-  absolute  depth  error 

0.28 

Characterization  Statistics  (TMA  Closest) 

#  Baseline 

#  Detected 

Pd 

#  Correct 

Pc° 

Ordnance 

43 

40 

0.93 

40 

1.00 

Nonordnance* 

78 

59 

0.76 

2 

0.03 

Large 

11 

11 

1.00 

5 

0.45 

Medium 

7 

7 

1.00  n 

7 

1.00 

Small 

25 

22 

0.88 

6 

0.27 

Abiflitv  to  Classify  _ _ _ 

Bomb 

21 

21 

1.00 

0 

0.00 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0  n 

NA 

Rocket 

22 

19 

0.86 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

"  Probability  of  detection 
'  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5. 1.6-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  1 :  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  Geophex 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5. 1.6-2 


GeoPotential  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA®Grou 


Ordnance  _ _ 


Nonordnance 


Total 


Number  False  Alarrns _ 


False  Alarm  Rate  (#/Hectare 


False  Alarm  Ratio  (#/Ord.) 


Probability  False  Alarms 


Localization  Statistics  (TMA  Closest)  (in  meters 


Mean  I  Std  Deviation 


dx  -  easting  error 


dv  -  northing  error 


Radiai  error  _ 


Depth  (z 


dz  -  averaged  depth  error 


idzl*^-  absolute  depth  error 


Characterization  Statistics  fTMA  Closest 


#  Baseline 


Ability  to  T 


Ordnance  _ 


Nonordnance 


Large 


Medium 


Smail 


Ability  to  Class! 


Bomb 


Projectile 


Mortar  _ 


Submunition 


Rocket 


#  Detected 


Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
®  Square  root  of  the  mean  square  depth  error 
^  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5. 1.7-1 


Depth  (meters) 


inia 


GRI  (Combined)  •  Aerial  Gunnery  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Pd“ 

Ordnance 

43 

41 

0.95 

Nonordnance 

77 

69 

0.90 

r 

Total 

120 

110 

Number  False  Alarms 

773 

False  Alarm  Rate  (#/Hectare) 

223.66 

False  Alarm  Ratio  (#/Orcl.) 

18.85 

Probability  False  Alarms 

0.281 

Localization  Statistics  (TM A  Closest )  (in  meters) 

Mean 

Std  Deviation  | 

. II 11 1^— 

dx  -  eastina  error 

0.02 

0.38 

dv  -  northing  error 

0 

0.38 

Radial  error 

0.43 

0.33 

iDepth  (z) _ _ _ J 

dz  -  averaged  depth  error 

0.13 

0.34  I 

Idzl'-  absolute  depth  error 

0.36  II 

Characterization  Statistics  (TMA  Closest) 

#  Baseline 

#  Detected 

Pd 

#  Correct 

Pc'* 

lAbilitv  to  Tvpe 

Ordnance 

43 

41 

0.95 

38 

0.93 

Nonordnance® 

78 

69 

0.88 

4 

0.06 

Large 

11 

11 

1.00 

7 

0.64 

Medium 

7 

7 

1.00 

0 

0.00 

Small 

25 

23 

0.92 

13 

0.57 

Abiiitv  to  Ciassify 

Bomb 

21 

21 

1.00 

10 

0.48 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

20 

0.91 

10 

0.50 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5.1.8-1 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  GRI-Combined 

Small  Medium 


■  Target  Detected 
□  Target  Not  Detected 


5. 1.8-2 


GRI  (EM)  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA*  Group) 


- 

#  Baseline 

#  Detected 

Ordnance 

43 

32 

0.74  1  0.138  1 

Nonordnance 

77 

62 

0.81 

Total 

120 

94 

Number  False  Alarms 

372 

False  Alarm  Rate  (#/Hectare) 

107.64 

False  Alarm  Ratio  (#/Ord.) 

11.63 

Probability  False  Alarms 

0.135 

Locaiization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  eastinq  error 

0.01 

0.41 

dv  -  northino  error 

0.14 

0.4 

Radial  error 

0.45 

0.38 

Depth  (z)  _ _ _ _ _ _ 

dz  -  averageci  depth  error 

-0.06 

0.23 

ldzl“-  absolute  depth  error 

0.24 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Pd 

#  Correct 

Pc^ 

Ordnance 

43 

32 

0.74 

32 

1.00 

Nonordnance® 

78 

62 

0.79 

2 

0.03 

Large 

11 

3 

0.27 

0 

0.00 

Medium 

7 

6 

0.86 

0 

0.00 

Small 

25 

23 

0.92 

0 

0.00 

Ability  to  Classify 

Bomb 

21 

13 

0.62 

0 

0.00 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0  1 

NA 

0 

NA 

Rocket 

22 

19 

0.86 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5.1.9-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  GRI-EM 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5. 1.9-2 


GRI  (Mag)  •  Aerial  Gunnery  Range 


Detection  Statistics  (TMA*  Group) 


Ordnance _ 

Nonordnance _ _ 

1======== 

Number  False  Alarms _ 

False  Alarm  Rate  (#/Hectare) 
False  Alarm  Ratio  (#/Ord.) 
Probability  False  Alarms _ 


#  Baseline 
43 
77 
120 


#  Detected 

38 

72 

110 


834 

241.31 

21.95 

0.303 


0.88 

0.94 


P  random 


Localization  Statistics  (TMA_C/gsggf}_(inmeters^ 


Position  (X.V) 


Mean 


Std  Deviation 


dx  -  easting  error 


dv  -  northing  error 


Radial  error 


0.03 


-0.03 


0.47 


0.4 


0.44 


0.36 


Depth  (z) 


0.14 


0.32 


Idzl“-  absolute  depth  error 

0.45 

Characterization  Statistics  (TMA  Closest) 

#  Baseline 

#  Detected 

pd 

#  Correct 

Pc" 

Ordnance 

43 

38 

0.88 

34 

0.89 

Nonordnance® 

78 

72 

0.92 

4 

0.06 

Larae 

11 

11 

1.00 

7 

Medium 

7 

7 

1.00 

0 

0,00 

Small 

25 

20 

0.80 

17 

0.85 

Abilitv  to  Classify 

Bomb 

21 

21 

1.00 

10 

0.48 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

17 

0.77 

12 

0.71 

Notes:  *  Target  Matching  Algorithm 

"  Probability  of  detection 
“  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
*TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5.1.10-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  1 :  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  GRI-Magnetometer 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.1.10-2 


NAEVA  •  Aerial  Gunnery  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

43 

38 

0.88  1  0.054  1 

Nonordnance 

77 

61 

0.79  1 

Total 

120 

99 

Number  False  Alarms 

113 

False  Alarm  Rate  (#/Hectare) 

32.70 

False  Alarm  Ratio  (#/Ord.) 

2.97 

Probability  False  Alarms 

0.041 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  easting  error 

-0.33 

0.36 

dy  •  northing  error 

0.03 

0.4 

Radial  error 

0.56 

0.29 

|Dept»l(z) _ _ _ _ _ 

dz  -  averaged  depth  error 

0.12 

0.24  I 

ldzl°-  absolute  depth  error 

0.28 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  #  Detected  |  Pd  #  Correct  Pc*’  I 

Ordnance 

43 

38 

0.88 

38 

1.00 

Nonordnance® 

78 

61 

0.78 

0 

0.00 

Ability  to  Size 

Large 

11 

11 

1.00 

11 

1.00 

Medium 

7 

5 

0.71 

0 

0.00 

Small 

25 

22 

0.88 

0 

0.00 

.  Ill  .  ^1^^— — — 

Bomb 

21 

20 

0.95 

20 

1.00 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

18 

0.82 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

“  Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5.1.11-1 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius;  2  meters 
Demonstrator:  NAEVA 

Small  Medium 


0.1 


S 

o 


£  1.0 


& 

o 

Q 


-I - 1 - 1 - 1 - ¥  I  I 


I  ■ 

f 


I 

■  I 

■  I 


10.0L 

10 


1  I  I  1  I  t  I  -  ’ 

100 

Principal  Diameter  (millimeters) 


■  Target  Detected 
□  Target  Not  Detected 


Lai 


5,1.11-2 


Rockwell  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

43 

23 

0.53  1  0.038  1 

Nonordnance 

77 

25 

0.32 

Total 

120 

48 

Number  False  Alarms 

84 

False  Alarm  Rate  (#/Hectare) 

24.30 

False  Alarm  Ratio  (#/Ord.) 

3.65 

Probability  False  Alarms 

0.031 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

iPosition  (x,v)  1 

dx  -  eastina  error 

0.09 

0.47 

dv  -  northina  error 

0.03 

0.78 

Radial  error 

0.76 

0.51 

Depth  (z) _ 

dz  •  averaged  depth  error 

-0.01  1  0.26  II 

Idzl®-  absolute  depth  error 

0.24 

Characterization  Statistics  (TMA  Closest) 


OllCllClwl.d  lAXmiwi  1  \ 

#  Baseline 

#  Detected 

Pd 

#  Correct 

Pc" 

lit  ni  K  nMi'l 

Ordnance 

43 

23 

0.53 

0 

0.00 

Nonordnance® 

78 

25 

0.32 

0 

0.00 

Larqe 

11 

9 

0.82 

0 

0.00 

Medium 

7 

4 

0.57 

0 

0.00 

Small 

25 

10 

0.40 

0 

0.00 

Ability  to  Classify 

Bomb 

21 

17 

0.81 

0 

0.00 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

6 

0.27 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5.1.12-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  Rockwell 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5-1.12-2 


SCA_ADI  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA^  Grou 


Notes:  ‘  Target  Matching  Algorithm 

”  Probability  of  detection 
®  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  •  not  applicable 


5.1.13-1 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  SCA_ADI 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.1.13-2 


SCA.Geo-Centers  -  Aerial  Gunnery  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Pd*^  I  Pranc^om 

Ordnance 

43 

40 

0.93  1  0.08 

Nonordnance 

77 

62 

0.81  i| 

Total 

120 

102 

Number  False  Alarms 

188 

False  Alarm  Rate  (#/Hectare) 

54.40 

False  Alarm  Ratio  (#/Ord.) 

4.70 

Probability  False  Alarms 

0.068 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

Position  (x,v) 

dx  -  eastinq  error 

-0.52 

0.58 

dv  -  northing  error 

0 

0.5 

Radial  error 

0.83 

0.41 

Deoth  (z) 

dz  -  averaged  depth  error 

-0.26 

0.31  1 

Idzl'-  absolute  depth  error 

0.41 

Characterization  Statistics  (TMA  Closest) 


. . — -T-V - 

#  Baseline 

#  Detected 

Pd 

#  Correct 

Pc- 

Ordnance 

43 

40 

0.93 

36 

0.90 

Nonordnance® 

78 

62 

0.79 

3 

0.05 

l>  1  III!  'll  1  II 1 

Large 

11 

11 

1.00 

4 

Medium 

7 

7 

1.00 

3 

0.43 

Small 

25 

22 

0.88 

18 

0.82 

Abilitv  to  Classify 

Bomb 

21 

19 

0.90 

0 

0.00 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

21 

0.95 

0 

0.00 

Notes:  *  T arget  Matching  Algorithm 

”  Probabiiity  of  detection 
”  Square  root  of  the  mean  square  depth  error 
“  Probabiiity  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


5.1.14-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  1:  Aerial  Gunnery  Range 
Critical  Radius:  2  meters 
Demonstrator:  SCA^Geocenters 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.1.14-2 


5.2  ARTILLERY  AND  MORTAR  RANGE 


This  section  presents  the  performance  of  the  following  demonstrators  that  participated  in  this  scenario: 


5.2.1 

ADI 

5.2.2 

Battelle 

5.2.3 

CHEMRAD 

5.2.4 

ENSCO 

5.2.5 

Geo-Centers 

5.2.6 

Geometries 

5.2.7 

Geophex 

5.2.8 

GeoPotential 

5.2.9 

GRI  (Combined) 

5.2.10 

GRI  (EM) 

5.2.11 

GRI  (Mag) 

5.2.12 

NAEVA 

5.2.13 

Rockwell 

5.2.14 

SCA_ADI 

5.2.15 

SCA_GeoCenters 

5.2.16 

SCA_Geometrics 

5.2-1 


ADI  •  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA°  Group) 


II 

#  Baseline 

#  Detected 

Pd^  I  ^  random 

Ordnance 

67 

57 

0.85  I  0.106 

Nonordnance 

50 

39 

0.78_J1 

Total 

117 

96 

Number  False  Alarms 

360 

False  Alarm  Rate  (#/Hectare) 

76.88 

False  Alarm  Ratio  (#/Ord.) 

6.32 

Probabiiity  False  Alarms 

0.097 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  eastinq  error 

0.03 

0.35 

dv  -  northing  error 

0.12 

0.42 

Radial  error 

0.43 

0.35 

Depth  (z) _ _ 

dz  -  averaged  depth  error 

0.17 

0.2  1 

Idzl®-  absolute  depth  error 

0.26  || 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Po“ 

#  Correct 

lAbilitv  to  Type 

Ordnance 

67 

57 

0.85 

55 

0.96 

Nonordnance 

50 

39 

0.78 

1 

0.03 

Larqe 

3 

3 

1.00 

3 

Medium 

31 

28 

0.90 

15 

0.54 

Small 

33 

26 

0.79 

2 

0.08 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

38 

0.93 

0 

0.00 

Mortar 

26 

19 

0.73 

7 

0.37 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  ‘  Target  Matching  Algorithm 

Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
NA  •  not  applicable 


5.2.1-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  HI 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  ADI 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.2el-2 


Battelle  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA^  Group,  5m  critical  radius) 


- - 

#  Baseline 

#  Detected 

Ordnance 

67 

8 

0.12  1  0.026  1 

Nonordnance 

50 

5 

0.10 

Total 

117 

13 

Number  False  Alarms 

8 

Feilse  Alarm  Rate  (#/Hectare) 

1.71 

False  Alarm  Ratio  (#/Ord.) 

1.00 

Probabilitv  False  Alarms 

0.013 

Localization  Statistics  (TMA  Closest,  5m  critical  radius)  (in  meters) 


Mean  I  Std  Deviation 


dx  -  easting  error 


dv  -  northing  error 


Radial  error 


-1.45 

2.24 

0.64 

2.39 

3.37 

1.07 

dz  -  averaged  depth  error 

0.57  1 

ldzl“-  absolute  depth  error 

0.98  1 

Characterization  Statistics  (TMA  Closest,  5m  critical  radius 


#  Baseline  I  #  Detected 


Ability  to  T 


Ordnance  I _ 67 _ _ 8 _ 


Nonordnance  I  50  I  5 


Large 


Medium 


Small 


Abilitv  to  Class! 


Bomb 


Projectile 


Mortar 


Submunition 


Rocket 


Notes:  “  Target  Matching  Algorithm 

**  Probability  of  detection 
"  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 


5.2.2- 1 


Depth  (meters) 


CHEMRAD  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

67 

29 

0.43  I  0.02  I 

Nonordnance 

50 

14 

0.28 

Total 

117 

23 

Number  False  Alarms 

48 

False  Alarm  Rate  (#/Hectare) 

10.25 

False  Alarm  Ratio  (#/Ord.) 

1.66 

Probability  False  Alarms 

0.013 

Locali^ionJtetisticsJTMA^tos^t)^im_mete^ 


Mean 

Std  Deviation  | 

iPosition  fx.v) 

dx  -  eastinq  error 

0.85 

0.65 

dv  -  northing  error 

-0.17 

0.61 

Radial  error 

1.14 

0.49 

Depth  (z)  _ _ _ _ _ _ 

dz  -  averaged  depth  error 

0.73 

0.44  J| 

ldzl“-  absolute  depth  error 

0.85 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  #  Detected  Pd“  #  Correct]  Pc  1 

Ordnance 

67 

29 

0.43 

0 

0.00 

Nonordnance 

50 

14 

0.28 

0 

0.00 

Large 

3 

2 

0.67 

0 

0.00 

Medium 

31 

12 

0.39 

0 

0.00 

Small 

33 

15 

0.45 

0 

0.00 

Abilitv  to  Classifv 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

20 

0.49 

0 

0.00 

Mortar 

26 

9 

0.35 

0 

0.00 

Submunition 

0 

0 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

"Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
"  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.3- 1 


Depth  (meters) 


ENSCO  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA^  Group) 


r  /  - 

#  Baseline 

#  Detected 

Ordnance 

67 

47 

0.70  I  0.065 

Nonordnance 

50 

28 

0.56 

Total 

117 

75 

Number  False  Alarms 

204 

False  Alarm  Rate  (#/Hectare) 

43.56 

False  Alarm  Ratio  (#/Ord.) 

4.34 

Probability  False  Alarms 

6.055 

Localization  Statistics  (TMA  C/osest)  (in  meters) 


Mean 

Std  Deviation  | 

. II II  I— ^—1— 

dx  -  eastina  error 

0.02 

0.25 

dy  -  northinq  error 

-0.04 

0.49 

Radial  error 

0.42 

0.36 

iDepth  (z)  _ _ _ 

dz  -  ayeragetl  depth  error 

•0.1 

0.2 | 

Idzl'-  absolute  depth  error 

0.22  || 

Characterization  Statistics  (TMACtoses^ 


wiiai  ^ 

#  Baseline 

#  Detected 

Pd“ 

#  Correct 

Pc^ 

IF  \  Hill'll  li'l ' 

Ordnance 

67 

47 

0.70 

0 

0.00 

Nonordnance 

50 

28 

0.56 

0 

0.00 

Large 

3 

3 

1.00 

0 

0.00 

Medium 

31 

22 

0.71  n 

0 

0.00 

Small 

33 

22 

0.67 

0 

0.00 

lAbilitv  to  Classify  - 1 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

33 

0.80 

0 

0.00 

Mortar 

26 

14 

0.54 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.4- 1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  ENSCO 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5. 2. 4-2 


Geo-Centers  •  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA®  Grou 


Ordnance  _ 


Nonordnance 


Total  


Number  False  Alarms 


False  Alarm  Rate  (#/Hectare) 


False  Alarm  Ratio  (#/Ord. 


Probability  False  Alarms 


#  Baseline 


67 


50 


117 


#  Detected 


62 


46 


108 


Localization  Statistics  (TMA  Closest)  (in  meters 


dx  -  easting  error 

0.07 

0.24 

dy  -  northing  error 

0.33 

0.29 

Radiai  error 

0.44 

0.24 

dz  •  averaged  depth  error 

0.38 

Idzl'-  absolute  depth  error 

0.45  1 

Characterization  Statistics  (TMA  Closest 


#  Baseline 


Ordnance 


Nonordnance 


Large 

3 

3 

1.00 

Medium 

31 

29 

0.94 

Small 

33 

30 

0.91 

Ability  to  Classi 


Bomb 


Projectile 


Mortar 


Submunition 


Rocket 


Notes:  "  T arget  Matching  Algorithm 

”  Probability  of  detection 
’’Square  root  of  the  mean  square  depth  error 
^  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.5-1 


P  r*  ^  P  P 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  GEO-CENTERS 


■  Target  Detected 
□  Target  Not  Detected 


Geometries  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA®  Grou 


Ordnance 


Nonordnance  _ 


Total  _ 


Number  False  Alarms 


False  Alarm  Rate  (#/Hectare 


False  Alarm  Ratio  (#/Ord. 


Probability  False  Alarms 


#  Baseline 


67 


50 


117 


180 


38.44 


3.00 


0.048 


Localization  Statistics  (TMA  Closest)  (in  meters 


dx  -  easting  error 

-0.01 

0.23 

dy  -  northing  error 

0.21 

0.38 

Radial  error 

0.39 

0.30 

dz  -  ayerageil  depth  error 

0.12 

Idzl'-  absolute  depth  error 

0.32  1 

Characterization  Statistics  (TMA  Closest 


#  Baseline 


Ability  to  T 


Ordnance  _ 


Nonordnance  _ 


Large  _ 


Medium 


Small  _ 


Ability  to  Class! 


Bomb  _ 


Projectile 


Mortar 


Submunition 


Rocket 


67 

60 

0.90 

59 

0.98 

50 

42 

0.84 

0 

0.00 

3 

3 

31 

29 

33 

28 

Notes:  ”  Target  Matching  Algorithm 

“  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.6- 1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  HI 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  Geometries 

Small  Medium  Lai 


m  Target  Detected 
□  Target  Not  Detected 


I 


Geophex  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA^  Group) 


#  Baseline 

#  Detected 

Ordnance 

67 

45 

0.67  1  0.038  1 

Nonordnance 

50 

30 

0.60  1 

Total 

117 

75 

Number  False  Alarms 

99 

False  Alarm  Rate  (#/Hectare) 

21.14 

False  Alarm  Ratio  (#/Ord.) 

2.20 

Probability  False  Alarms 

0.027 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation 

iPosition  (x.v) 

dx  -  easting  error 

0.02 

0.44 

dy  -  northing  error 

0.22 

0.36 

Radial  error 

0.51 

0.34 

iDepth  (z)  _ _ _ _ 

dz  -  averaged  depth  error 

0.05 

0.24  1 

Idzl®-  absolute  depth  error 

0.24  1 

Characterization  Statistics  (TMA  Closes^ 


#  Baseline 

#  Detected 

Pd“ 

#  Correct 

Pc- 

Ordnance 

67 

45 

0.67 

45 

1.00 

Nonordnance 

50 

30 

0.60 

0 

0.00 

Large 

3 

3 

1.00 

0 

Medium 

31 

20 

0.65 

9  1 

0.45 

Small 

33 

22 

0.67 

16 

0.73 

AbiMtv  to  Classify 

Bomb 

0 

0 

NA 

0 

NA 

Proiectile 

41 

30 

0.73 

0  1 

0.00 

Mortar 

26 

15 

0.58 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  '  Target  Matching  Algorithm 

"  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.7-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  HI 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  Geophex 

Small  Medium  Large 


■  Target  Detected 
a  Target  Not  Detected 


5. 2. 7-2 


GeoPotential  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TM A°  Group ) 


#  Baseline 

#  Detected 

Pd“ 

Ordnance 

67 

2 

0.03 

0.006 

Nonordnance 

50 

1 

0.02 

1 

Total 

117 

3 

Number  Faise  Alarms 

20 

False  Alarm  Rate  (#/Hectare) 

4.27 

False  Alarm  Ratio  (#/Ord.) 

10.00 

Probability  False  Alarms 

0.005 

Locaiization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation 

iPosition  (x,v) 

dx  -  easting  error 

-0.4 

0.73 

dy  -  northing  error 

-0.16 

0.54 

Radiai  error 

0.81 

0.32 

iDepth  (z) _ 

dz  •  averaged  depth  error 

0.24 

0.21 

idzl®-  absolute  depth  error 

0.30 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  #  Detected  Pd”  #  Correct  |  Pc“  I 

\}  \  Hill'll  ii'i ' 

Ordnance 

67 

2 

0.03 

2 

1.00 

Nonordnance 

50 

1 

0.02 

0 

0.00 

Large 

3 

0 

0.00 

0 

Medium 

31 

1 

0.03 

1 

1.00 

Small 

33 

1 

0.03 

0 

0.00 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

1 

0.02 

0 

0.00 

Mortar 

26 

1 

0.04 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Aigorithm 

”  Probabiiity  of  detection 
°  Square  root  of  the  mean  square  depth  error 
"  Probabiiity  of  correct  characterization 
NA  -  not  appiicabie 


5.2.8-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  GeoPotenttal 

Small  Medium 


■  Target  Detected 
a  Target  Not  Detected 


5. 2. 8-2 


GRI-Combined  •  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA^  Group) 


■  — r-  7 

#  Baseline 

Ordnance 

67 

60  0.90  1  0.289  | 

Nonordnance 

50 

50  1 

1.00  1 

Total 

117 

110 

Number  False  Aianms 

1209 

False  Alarm  Rate  (#/Hectare) 

258.18 

False  Alarm  Ratio  {#/Ord.) 

20.15 

Probabiiitv  False  Alarms 

0.324 

Localization  Statistics  (TMA  Closest)  (in  metergl 


Mean 

■ncifiiamHji 

dx  -  eastino  error 

-0.05 

0.22 

dv  -  northinq  error 

0.1 

0.35 

Radial  error 

0.37 

0.21 

Deothfz)  1 

dz  •  averaged  depth  error 

0.15 

0.32  1 

ldzl‘-  absolute  depth  error 

0.35 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Pd“ 

#  Correct 

Pc'- 

Abilitv  to  Tvpe  _ _ — , 

Ordnance 

67 

60 

0.90 

58 

0.97 

Nonordnance 

50 

50 

1.00 

2 

0.04 

Larqe 

3 

3 

1.00 

0 

0.00 

Medium 

31 

29 

0.94 

7 

0.24 

Small 

33 

28 

0.85 

15 

0.54 

Abilitv  to  Classify 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

38 

0.93 

11 

0.29 

Mortar 

26 

22 

0.85 

10 

0.45 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
NA  -  not  appiicable 


5.2.9- 1 


Depth  (meters) 


GRI-EM  •  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA^  Group) 


#  Baseline 

#  Detected 

Pd“ 

P  random 

Ordnance 

67 

58 

0.87 

0.151  1 

Nonordnance 

50 

49 

0.98 

1 

Total 

117 

107 

Number  False  Alarms 

549 

False  Alarm  Rate  (#/Hectare) 

117.24 

False  Alarm  Ratio  (#/Ord.) 

9.47 

Probability  False  Alarms 

0.147 

Localization  Statistics  (TMA  Closest)  (in  meters) 


1  Mean  Std  Deviation  |] 

dx  -  eastinq  error 

-0.06 

0.25 

dv  -  northing  error 

0.02 

0.32 

Radial  error 

0.38 

0.16 

iDepth  (z) _ _ 

dz  -  average^  depth  error 

0.28 

0.22  II 

ldzl‘-  absolute  depth  error 

0.36 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Po*> 

#  Correct 

Pc- 

Ordnance 

67 

58 

0.87 

57 

0.98 

Nonordnance 

50 

49 

0.98 

0 

0.00 

Large 

3 

3 

1.00 

0 

0.00 

Medium 

31 

27 

0.87 

0 

0.00 

Small 

33 

28 

0.85 

0 

0.00 

Ability  to  Classify 

Bomb 

0 

0 

NA 

0 

NA 

Proiectiie 

41 

36 

0.88 

0 

0.00 

Mortar 

26 

22 

0.85 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  "  Target  Matching  Algorithm 

Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.10-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  GRI-EM 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.2.10-2 


GRI-Mag  •  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA^  Group) 


#  Baseline 

#  Detected 

Ordnance 

67 

62 

0.93  1  0.27 

Nonordnance 

50 

41 

0.82  ! 

Total 

117 

103 

Number  False  Alarms 

1111 

False  Alarm  Rate  (#/Hectare) 

237.25 

False  Alarm  Ratio  (#/Ord.) 

17.92 

Probability  False  Alarms 

0.298 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  eastinq  error 

-0.08 

0.31 

dy  -  northing  error 

0.18 

0.43 

Radial  error 

0.46 

0.32 

Depth  (z) 

dz  -  averaged  depth  error 

0.08 

0.37  1 

IdzP-  absolute  depth  error 

0.37  .11 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  #  Detected  Pd“  |#  Correct  Pc° 

Ordnance 

67 

62 

0.93 

60 

0.97 

Nonordnance 

50 

41 

0.82 

3 

0.07 

Large 

3 

3 

1.00 

0 

0.00 

Medium 

31 

28 

0.90  n 

8 

0.29 

Small 

33 

31 

0.94 

25 

0.81 

lAbilitv  to  Classify  1 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

38 

0.93 

14 

0.37 

Mortar 

26 

24 

0.92 

20 

0.83 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.11-1 


Depth  (meters) 


■  Target  Detected 
a  Target  Not  Detected 


NAEVA  >  Artillery  and  Mortar  Range 

Detection  Statistics  (TMA®  Group) 

#  Baseline 

#  Detected 

Ordnance 

67 

65 

0.97  1  0.041  1 

Nonordnance 

50 

48 

0.96  1 

Total 

117 

113 

Number  False  Alarms 

89 

False  Alarm  Rate  (#/Hectare) 

19.01 

False  Alarm  Ratio  (#/Ord.) 

1.37 

Probability  False  Alarms 

0.024 

Localization  Statistics  (TMA  Closest)  (in  meters) 


1 

Mean 

Std  Deyiation  | 

dx  -  eastinq  error 

-0.24 

0.46 

dv  -  northinq  error 

0.12 

0.43 

Radial  error 

0.59 

0.34 

Deoth  (z) 

dz  -  averagefi  depth  error 

0.2 

0.22  1 

Idzl'-  absolute  depth  error 

0.30  II 

Characterization  Statistics  (TMA  Closest) 


67 

65 

0.97 

65 

1.( 

Nonordnance 

50 

48 

0.96 

0 

0.( 

#  Baseline 


#  Detected 


Pp**  #  Correct  |  Pc 


Larqe 

3 

3 

1.00 

0 

0. 

Medium 

31 

29 

0.94 

29 

1. 

Small 

33 

33 

1.00 

0 

0. 

Ability  to  Classify 


Bomb 


Projectile 


Mortar 


Submunition 


Rocket 


41 


26 


39 


26 


NA 


0.95 


1.00 


NA 


NA 


39 


_± 

_L 

0. 


NA 


NA 


Notes:  ”  Target  Matching  Algorithm 

^  Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
^  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.12-1 


§§>  §§8  8§ 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  NAEVA 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.2.12-2 


Rockwell  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA^  Grou 


Ordnance _ 


Nonordnance  _ 


Total  _ _ 


Number  False  Alamris _ _ 


False  Alarm  Rate  (#/Hectare) 


False  Alarm  Ratio  (#/Ord. 


Probability  False  Alarms 


#  Baseline 

#  Detected 

67 

14 

50 

10 

Larqe 


Medium 


Small 


Ability  to  Classi 


Bomb 


Projectile 


Mortar 


Submunition 


Rocket 


Notes:  "  Target  Matching  Algorithm 

Probability  of  detection 
®  Square  root  of  the  mean  square  depth  error 
^  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.13-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  Rockwell 


Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.2.13-2 


SCA.ADI  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA^  Group) 


i  /  - 

#  Baseline 

#  Detected 

Pd“ 

P  random 

Ordnance 

67 

55 

0.82 

0.058  1 

Nonordnance 

50 

29 

0.58 

I 

Total 

117 

84 

Number  False  Alarms 

168 

False  Alarm  Rate  (#/Hectare) 

35.88 

False  Alarm  Ratio  (#/Ord.) 

3.05 

Probability  False  Alarms 

0.045 

Localization  Statistics  (TMA  Closest)  (in  meterej 


Mean 

Std  Deviation  | 

dx  -  easting  error 

-0.07 

0.45 

dy  -  northing  error 

-0.21 

0.44 

Radial  error 

0.57 

0.33 

Depth  (z) _ _ _ _ _ 

dz  -  ayeraged  depth  error 

-0.08 

0.22  1 

ldzl°-  absolute  depth  error 

0.22  || 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  #  Detected  Po° 

#  Correct  |  Pc'* 

Ordnance 

67 

55 

0.82 

51 

0.93 

Nonordnance 

50 

29 

0.58 

0 

0.00 

Large 

3 

3 

1.00 

0 

0.00 

Medium 

31 

27 

0.87 

13 

0.48 

Small 

33 

_ 25  _ 

0.76 

22 

0.88 

lAbllitv  to  Classify  J 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

37 

0.90 

0 

0.00 

Mortar 

26 

2 

0.08 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Aigorithm 

”  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.14-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  SCA_ADI 

Small  Medium  Large 


■  Target  Detected 
a  Target  Not  Detected 


5.2.14-2 


SCA_Geocenters  -  Artillery  and  Mortar  Range 


Detection  Statistics  (TMA*  Group) 


- - - 

#  Baseline 

#  Detected 

Po“ 

Ordnance 

67 

51 

0.76 

0.06 

Nonordnance 

50 

31 

0.62 

1 

Total 

117 

82 

Number  False  Alarms 

178 

False  Alarm  Rate  (#/Hectare) 

38.01 

False  Alarm  Ratio  (#/Ord.) 

3.49 

Probability  False  Alarms 

0.048 

Localization  Statistics  (TMA  Closest)  (in  meters) 


1 

Mean 

Std  Deviation 

Position  fx.vl 

dx  -  eastina  error 

0.05 

0.49 

dy  -  northinq  error 

0.28 

0.52 

Radial  error 

0.65 

0.39 

Depth  (z) _ _ _ 

dz  •  averagefi  depth  error 

0.02 

0.29 

Idzl'-  absolute  depth  error 

0.30 

Characterization  Statistics  (TMA  Closest) 


Wl  lai  wai  ^ 

#  Baseline 

#  Detected 

Pd“ 

#  Correct 

Pc- 

Ordnance 

67 

51 

0.76 

44 

0.86 

Nonordnance 

50 

31 

0.62 

0 

0.00 

Large 

3 

3 

1.00 

1 

0.33 

Medium 

31 

28 

0.90 

9 

0.32 

Small 

33 

20 

0.61 

16 

0.80 

Abilitv  to  Classify 

Bomb 

0 

0 

NA 

0 

NA 

Proiectile 

41 

36 

0.88 

0 

0.00 

Mortar 

26 

15 

0.58 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
‘  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.15-1 


JPG  Controlled  Site;  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius;  2  meters 
Demonstrator:  SCA_Geocenters 

Small  Medium 


Principal  Diameter  (millimeters) 


■  Target  Detected 
□  Target  Not  Detected 


5.2.15-2 


SCA_Geometrics  -  Artillery  and  Mortar  Range 


netection  Statistics  (TMA®  Gfooo) _ _ _  -i 

#  Baseline 

#  Detected 

Ordnance 

67 

64 

0.96  1  0.068  11 

Nonordnance 

50 

46 

0.92  1 

Total 

117 

110 

Number  False  Alarms 

196 

False  Alarm  Rate  (#/Hectare) 

41.86 

False  Alarm  Ratio  (#/Ord.) 

3.06 

Probability  False  Alarms 

0.053 

Localization  Statistics  (TMA  Closest)  (in  meters) 


I 


Mean 


Position  (x,v) 


dx  •  easting  error 


dv  -  northing  error 


Radial  error 


-0.09 


0.11 


0.49 


Depth  (z) 


dz  -  averaged  depth  error 


-0.04 


Idzl®  -  absolute  depth  error 

0.26  1 

nhflracterization  Statistics  tTMA  Closest) 

#  Baseline 

#  Detected 

Po“ 

#  Correct 

Pc 

—  ir  - 

Ordnance 

67 

64 

0.96 

58 

0.91 

Nonordnance 

50 

46 

0.92 

3 

0.07 

3 

3 

1.00 

2 

0.67 

Medium 

31 

29 

0.94 

18 

0.62 

Small 

33 

32 

0.97 

18 

0.56 

AhMitv  to  Classifv  _ _  _ _ 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

39 

0.95 

0 

0.00 

Mortar 

26 

25 

0.96 

0 

0.00 

Submunition 

0 

0  n 

NA  ^ 

0  n 

NA 

Rocket 

0 

0 

NA 

0  n 

NA 

Notes:  *  Target  Matching  Algorithm 

Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
"  Probability  of  correct  characterization 
NA  -  not  applicable 


5.2.16-1 


JPG  Controlled  Site:  Phase  III 
Scenario  2:  Artillery  and  Mortar  Range 
Critical  Radius:  2  meters 
Demonstrator:  SCA^Geometrlcs 

Small  Medium 


■  Target  Detected 
□  Target  Not  Detected 


5.2.16-2 


5.3  GRENADE  AND  SUBMUNITION  RANGE 


This  section  presents  the  performance  of  the  following  demonstrators  that  participated  in  this  scenario 


5.3.1 

ADI 

5.3.2 

Geo-Centers 

5.3.3 

GeoPotential 

5.3.4 

GRI  (Combined) 

5.3.5 

GRI(EM) 

5.3.6 

GRI  (Mag) 

5.3.7 

SCA.ADI 

5.3.8 

SCA_GeoCenters 

5.3-1 


ADI  -  Grenade  and  Submunition  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

98 

71 

0.72  1  0.191  1 

Nonordnance 

39 

27 

0.69 

Total 

137 

98 

Number  False  Alarms 

625 

False  Alarm  Rate  (#/Hectare) 

150.70 

False  Alarm  Ratio  (#/Ord.) 

8.80 

Probability  False  Alarms 

0.189 

Locaiization  Statistics  (TMA  Closest)  (in  meters) 


_ 

Mean 

dx  -  easting  error 

0.01 

0.37 

dy  -  northing  error 

0.12 

0.48 

Radial  error 

0.48 

0.38 

dz  •  averaged  depth  error 

0.32 

0.14  1 

ldzl‘-  absolute  depth  error 

0.36 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  #  Detected  Po*’  #  Correct  Pc‘  | 

Ordnance 

98 

71 

0.72 

41 

0.58 

Nonordnance 

39 

27 

0.69 

2 

0.07 

lAbilitv  to  Size  \ 

Large 

0 

0 

NA 

0 

NA 

Medium 

1 

1 

1.00 

0 

0.00 

Small 

97 

70 

0.72 

68 

0.97 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

1 

1 

1.00 

0 

0.00 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

97 

70 

0.72 

64 

0.91 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  '  Target  Matching  Aigorithm 

”  Probabiiity  of  detection 
‘Square  root  of  the  mean  square  depth  error 
‘Probability  of  correct  characterization 
NA  •  not  applicable 


5.3.1-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 

Scenario  3:  Grenade  and  Submunitlons  Range 

Critical  Radius:  2  meters 

Demonstrator:  ADI 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.3. 1-2 


Geo-Centers  •  Grenade  and  Submunition  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

98 

89 

0.91  1  0.127 

Nonordnance 

39 

24 

0.62 

Total 

137 

113 

Number  False  Alanns 

356 

False  Alarm  Rate  (#/Hectare) 

85.84 

False  Alarm  Ratio  (#/Ord.) 

4.00 

Probability  False  Alarms 

0.108 

Localization  Statistics  (TMA  Closest)  (in  meters) 


_ 

Mean 

Std  Deviation  | 

dx  -  easting  error 

-0.25 

0.4 

dy  -  northing  error 

0.3 

0.43 

Radial  error 

0.62 

0.34 

dz  -  averaged  depth  error 

0.17 

0.16  1 

ldzl‘-  absolute  depth  error 

0.24 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  |  #  Detected  |  Pd”  #  Correct  Pc‘ 

Ability  to  Type 

Ordnance 

98 

89 

0.91 

89 

1.00 

Nonordnance 

39 

24 

0.62 

0 

0.00 

lAbllity  to  Size  1 

0 

0 

NA 

0 

NA 

Medium 

1 

1 

1.00 

1 

1.00 

Small 

97  I 

88 

0.91 

87 

0.99 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

1 

1 

1.00 

0 

0.00 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

97 

88 

0.91 

88 

1.00 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
‘Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
NA  -  not  applicable 


5.3.2-1 


Depth  (meters) 


GeoPotential  -  Grenade  and  Submunition  Range 


Detection  Statistics  (TMA®  Grqu 


Ordnance  _ 


Nonordnance  _ 


Total  


Number  False  Alarms 


False  Alarm  Rate  (#/Hectare 


False  Alarm  Ratio  (#/Ord. 


Probability  False  Alarms 


#  Baseline 


98 


39 


137 


#  Detected 


1 


0 


1 


Localization  Statistics  (TMA  Closest)  (In  meters 


Position  (X 


dx  -  eastino  error 


dv  -  northing  error 


Radial  error 


Std  Deviation 


dz  -  averaged  depth  error 

0.78  1 

Idzl'-  absolute  depth  error 

0.78 

Characterization  Statistics  (TMA  Closest 


#  Baseline 


Ordnance 


Nonordnance 


Ability  to  Class! 


Bomb 


Projectile 


Mortar 


Submunition 


Rocket  


#  Detected 


#  Correct 


Large 

0 

0 

NA 

Medium 

1 

1 

1.00 

Small 

97 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

"  Probability  of  detection 
”  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
NA  -  not  applicable 


5.3.3-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  111 

Scenario  3:  Grenade  and  Submunltlons  Range 

Critical  Radius:  2  meters 

Demonstrator:  GeoPotentlai 

Small  Medium  Urge 


■  Target  Detected 
□  Target  Not  Detected 


5. 3. 3-2 


GRI  (Combined)  •  Grenade  and  Submunition  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Po“ 

Ordnance 

98 

93 

0.95 

0.277 

Nonordnance 

39 

28 

0.72 

1 

Total 

137 

121 

Number  Feilse  Alarms 

973 

False  Alarm  Rate  (#/Hectare) 

234.62 

False  Alarm  Ratio  (#/Ord.) 

10.46 

Probability  False  Alarms 

0.295 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  eastinq  error 

-0.08 

0.29 

dy-  northing  error 

0.13 

0.32 

Radial  error 

0.37 

0.26 

iDepthJz) _ J 

dz  -  averaged  depth  error 

0.39 

0.32  I 

Idzl'-  absolute  depth  error 

0.50  1 

CharacterizrtionJ^|stics_gMA_^^ws^ 


#  Baseline 

#  Detected 

Po** 

#  Correct 

Pc- 

Ordnance 

98 

93 

0.95 

86 

0.92 

Nonordnance 

39 

28 

0.72 

10 

0.36 

Large 

0 

0 

NA 

0 

NA 

Medium 

1 

1 

1.00 

0  n 

0.00 

Small 

97 

92 

0.95 

32 

0.35 

Ability  to  Classify  - 1 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

1 

1 

1.00 

0 

0.00 

Mortar 

0 

0 

NA 

0 

NA 

Submunltlon 

97 

92 

0.95 

0 

0.00 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
Probability  of  correct  characterization 
NA  -  not  applicable 


5.3.4-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 

Scenario  3:  Grenade  and  Submunitions  Range 

Critical  Radius:  2  meters 

Demonstrator:  GRI-Combined 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5. 3. 4-2 


GRI  (EM)  •  Grenade  and  Submunition  Range 


Detection  Statistics  (TMA^  Group) 


#  Baseline 

#  Detected 

II^HZii[|liiE2!SS!l 

Ordnance 

98 

90 

0.92  1  0.19 

Nonordnance 

39 

24 

0.62 

Total 

137 

114 

Number  False  Alarms 

601 

False  Alarm  Rate  (#/Hectare) 

144.92 

False  Alarm  Ratio  (#/Ord.) 

6.68 

Probability  False  Alarms 

0.182 

Localization  Statistics  (TMA  Closest)  (in  meters) 


_ 

Mean 

Std  Deviation  | 

dx  -  eastinq  error 

-0.06 

0.29 

dy  -  northing  error 

0.15 

0.26 

Radial  error 

0.34 

0.25 

dz  -  averageq  depth  error 

0.67 

0.13  I 

Idzl®-  absolute  depth  error 

0.69 

Characterization  Statistics  (TMA  Closest) 


I  #  Baseline  #  Detected  Pd”  |#  Correct  Pc” 

1 1  lllii'il  ii'i  I  I 

Ordnance 

98 

90 

0.92 

89 

0.99 

Nonordnance 

39 

24 

0.62 

5 

0.21 

lAbiiity  to  Size  1 

Large 

0 

0 

NA 

0 

NA 

Medium 

1 

1 

1.00 

0 

0.00 

Small 

97 

89 

0.92 

0 

0.00 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

1 

1 

1,00 

0 

0.00 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

97 

89 

0.92 

0 

0.00 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  ”  Target  Matching  Algorithm 

"  Probability  of  detection 
’’Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
NA  -  not  applicable 


5.3.5-1 


GRI  (Mag)  -  Grenade  and  Submunition  Range 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

98 

46 

0.47  1  0.228 

Nonordnance 

39 

27 

0.69 

Total 

137 

73 

Number  False  Alarms 

803 

False  Alarm  Rate  (#/Hectare) 

193.62 

False  Alarm  Ratio  (#/Ord.) 

17.46 

Probability  False  Alarms 

0.243 

Localization  Statistics  (TMA  Closest)  (in  meters) 


_ 

Mean 

Std  Deviation  | 

dx  -  eastinq  error 

-0.18 

0.38 

dv  -  northing  error 

0.12 

0.5 

Radial  error 

0.54 

0.38 

iDepthfz) _ _ _ _ _ 

dz  •  averaged  depth  error 

0.23 

0-38  I 

Idzl'-  absolute  depth  error 

0.45 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Po“ 

#  Correct 

Pc"  1 

Ordnance 

98 

46 

0.47 

34 

0.74 

Nonordnance 

39 

27 

0.69 

9  ^ 

0.33 

lAbllitv  to  Size  _ I 

Large 

0 

0 

NA 

0 

NA 

Medium 

1 

1 

1.00 

0 

0.00 

Small 

97 

45 

0.46 

45 

1.00 

lAbilitv  to  Classify  I 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

1 

1 

1.00 

0 

0.00 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

97 

45 

0.46 

0 

0.00 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

"Probability  of  detection 
"Square  root  of  the  mean  square  depth  error 
“Probability  of  correct  characterization 
NA  -  not  applicable 


5.3.6-1 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 

Scenario  3:  Grenade  and  Submunitions  Range 

Crfticai  Radius:  2  meters 

Demonstrator:  GRi-Magnetometer 

Small  Medium  Large 


Principal  Diameter  (millimeters) 


■  Target  Detected 
□  Target  Not  Detected 


5. 3. 6-2 


SCA_ADI  -  Grenade  and  Submunition  Range 


Detection  Statistics  (TMA^  Group) 


#  Baseline 

#  Detected 

Ordnance 

98 

45 

0.46  1  0,081 

Nonordnance 

39 

16 

0.41 

Total 

137 

61 

Number  False  Alarnis 

232 

False  Alarm  Rate  (#/Hectare) 

55.94 

False  Alarm  Ratio  (#/Ord.) 

5.16 

Probability  False  Alarms 

0.070 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  eastinq  error 

0.04 

0.46 

dy  -  northing  error 

0 

0.46 

Radial  error 

0.51 

0.40 

dz  -  averaged  depth  error 

0.08 

0.06  I 

Idzi®-  absolute  depth  error 

0.10 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  #  Detected  Pd*’  #  Correct 

mem 

Ordnance 

98 

45 

0.46 

44 

0.98 

Nonordnance 

39 

16 

0.41 

0 

0.00 

lAbilitv  to  Size  J 

Large 

0 

0 

NA 

0 

NA 

Medium 

1 

1 

1.00 

0 

0.00 

Small 

97  1 

44 

0.45 

44 

1.00 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

1 

1 

1.00 

0 

0.00 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

97 

44 

0.45 

43 

0.98 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

**  Probability  of  detection 
”  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
NA  -  not  applicable 


5.3.7-1 


JPG  Controlled  Site;  Phase  ill 
Scenario  3:  Grenade  and  Submunitions  Range 
Critical  Radius:  2  meters 
Demonstrator:  SCA^ADI 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


5.3. 7-2 


SCA_Geo-Centers  •  Grenade  and  Submunition  Range 


Detection  Statistics  (TMA^  Group) 


#  Baseline 

#  Detected 

Ordnance 

98 

68 

0.69  1  0.07 

Nonordnance 

39 

19 

0.49 

Total 

137 

87 

Number  False  Alarms 

169 

False  Alarm  Rate  (#/Hectare) 

40.75 

False  Alarm  Ratio  (#/Ord.) 

2.49 

Probability  False  Alarms 

0.051 

Locaiization  Statistics  (TMA  Closest)  (in  meters) 


_ 

Mean 

Std  Deviation  | 

iPosition  (x,y) 

dx  -  easting  error 

-0.28 

0.47 

dv  -  northing  error 

0.44 

0.44 

Radial  error 

0.71 

0.41 

dz  •  averaged,  depth  error 

0.8 

NA  I 

Idzl'-  absolute  depth  error 

0.80 

Characterization  Statistics  (TMA  Closest) 


,  "  - -  \  ,  -■  h 

#  Baseline  #  Detected  Pd 

#  Correct  Pc“ 

Ordnance 

98 

68 

0.69 

67 

0.99 

Nonordnance 

39 

19 

0.49 

0 

0.00 

lAbilitv  to  Size  I 

Large 

0 

0 

NA 

0 

NA 

Medium 

1 

1 

1.00 

1 

1.00 

Small 

97 

67 

0.69 

67 

1.00 

lAbllitv  to  Classify  I 

Bomb 

0 

0 

NA 

0 

NA 

Proiectile 

1 

1 

1.00 

0 

0.00 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

97 

67 

0.69 

67 

1.00 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  '  Target  Matching  Algorithm 

^  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
NA  -  not  applicable 


5.3.8-1 


JPG  Controlled  Site:  Phase  III 

Scenario  3:  Grenade  and  Submunitions  Range 

Critical  Radius:  2  meters 

Demonstrator:  SCA^Geocenters 

Small  Medium 


■  Target  Detected 
□  Target  Not  Detected 


5.4  INTERROGATION/BURIAL  AREA 


This  section  presents  the  performance  of  the  following  demonstrators  that  participated  in  this  scenario; 


5.4.1 

ADI 

5.4.2 

Battelle 

5.4.3 

ENSCO 

5.4.4 

Foerster 

5.4.5 

Geometries 

5.4.6 

SCA.ADI 

5.4.7 

SCA_Geometrics 

5.4-1 


ADI  -  Interrogation/Burial  Area 


Localization  Statistics  (in  meters) 


Mean 

Std  Deviation 

Position  (x,y) 

dx  -  easting  error 

-0.03 

0.2 

dy  -  northing  error 

0.11 

0.2 

Radial  error 

0.26 

0.14 

dz  -  averaged  depth  error 

0.25 

0.34 

Idzl®-  absolute  depth  error 

0.42 

Characterization  Statistics 


1  #  Baseline  #  Correct 

Pc*- 

Ability  to  Type 

Ordnance 

17 

17 

1.00 

Nonordnance 

3 

_ Q _ 

0.00 

Ability  to  Size 

Large 

5 

5 

1.00 

Medium  ^ 

7 

2 

0.29 

Smail 

5 

0 

0.00 

Bomb 

4 

4 

1.00 

Projectile 

9 

0 

0.00 

Mortar 

4 

2 

0.50 

Submunition 

0 

0 

NA 

Rocket 

0 

0 

NA 

Notes:  “  Square  root  of  the  mean  square  depth  error 

”  Probabiiity  of  correct  characterization 


5.4. 1-1 


Battelle  -  Interrogation/Burial  Area 


Localization  Statistics  (in  meters) 


Mean 

Std  Deviation  | 

iPosition  (x,y)  II 

dx  -  easting  error 

-0.3 

0.7 

dy  -  northing  error 

0.36 

0.47 

Radiai  error 

0.77 

0.54 

iDepth  (z) _ 

dz  -  averaged  depth  error 

0.14 

0.98  I 

Idzl®  -  absoiute  depth  error 

0.92 

Characterization  Statistics 


#  Baseiine  #  Correct  Pc° 

lAbility  to  Type 

Ordnance 

4 

0 

0.00 

Nonordnance 

4 

0 

0.00 

lAbiiity  to  Size  II 

Large 

3 

0 

0.00 

Medium 

1 

0 

0.00 

Smali 

0 

0 

N/A 

Bomb 

3 

0 

0.00 

Projectile 

1 

0 

0.00 

Mortar 

0 

0 

N/A 

Submunition 

0 

0 

NA 

Rocket 

0 

0 

NA 

Notes:  *  Square  root  of  the  mean  square  depth  error 

Probability  of  correct  characterization 


5.4.2 -1 


ENSCO  -  Interrogation/Burial  Area 


Localization  Statistics  (in  meters) 


Mean  |  Std  Deviation  | 

Position  (x,y) 

dx  -  easting  error 

0.06 

0.24 

dy  -  northing  error 

-0.08 

0.26 

Radial  error 

0.30 

0.20 

Depth  (z)  1 

dz  -  averaged  depth  error 

0.07  0.36  II 

Idzl®  -  absolute  depth  error 

0.36 

Characterization  Statistics 


#  Baseline 

#  Correct 

Pc“ 

lAbility  to  Type  11 

Ordnance 

17 

14 

0.82 

Nonordnance 

3 

0 

0.00 

lAbilitv  to  Size 

Large 

5 

2 

0.40 

Medium 

7 

2 

0.29 

Small 

5 

4 

0.80 

lAbility  to  Classify 

Bomb 

4 

0 

0.00 

Projectile 

10 

0 

0.00 

Mortar 

3 

0 

0.00 

Submunition 

0 

0 

NA 

Rocket 

0 

0 

NA 

Notes:  *  Square  root  of  the  mean  square  depth  error 

*’  Probability  of  correct  characterization 


5.4.3 -1 


Foerster  -  Interrogation/Burial  Area 


Localization  Statistics  (in  meters) 


Mean 

Std  DeviationH 

Position  (x,y) 

dx  -  easting  error 

0.06 

0.64 

dy  •  northing  error 

-0.01 

0.48 

Radial  error 

0.52 

0.60 

iDepth  (z) _ _ _ II 

dz  -  averaged  depth  error 

0.39 

0.73 

Idzl®-  absolute  depth  error 

0.81 

Characterization  Statistics 


#  Baseline 

#  Correct  Pc**  I 

lAbility  to  Type  II 

Ordnance 

19 

0 

0.00 

Nonordnance 

_ 5 _ 1 

0 

0.00 

lAbility  to  Size  II 

Large  ^ 

6 

6 

1.00 

Medium 

6 

4 

0.67 

Small 

7 

0 

0.00 

Bomb 

5 

0 

0.00 

Projectile 

10 

0 

0.00 

Mortar 

4 

0 

0.00 

Submunition 

0 

0 

NA 

Rocket 

0 

0 

NA 

Notes:  ‘  Square  root  of  the  mean  square  depth  error 

Probability  of  correct  characterization 


5.4.4 -1 


Geometries  •  Interrogation/Buriai  Area 


Localization  Statistics  (in  meters)* 


Mean 

Std  Deviation 

Position  (x,y) 

dx  -  easting  error 

-0.08 

0.09 

dy  -  northing  error 

-0.07 

0.13 

Radial  error 

0.16 

0.09 

iDepth  (z) _ _ _ 

dz  -  averaged  depth  error 

0.23  0.43  1 

Idzl**  -  absolute  depth  error 

0.48 

Characterization  Statistics* 


#  Baseline 

#  Correct 

Pc'- 

lAbility  to  Type 

Ordnance 

10 

10 

1.00 

Nonordnance 

3 

0 

0.00 

Ability  to  Size 

Large 

3 

2 

0.67 

Medium  ^ 

3 

2 

0.67 

Small 

4 

4 

1.00 

Bomb 

2 

2 

1.00 

Projectile 

6 

4 

0.67 

Mortar 

2 

1 

0.50 

Submunition 

0 

0 

NA 

Rocket 

0 

0 

NA 

Notes: 

*Only  13  of  20  reported  targets  positions  corresponded 
to  baseline  targets  at  Rcrit  of  2  meters. 

Square  root  of  the  mean  square  depth  error 
Probability  of  correct  characterization 
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SCA_ADI  -  Interrogation/Burial  Area 


Localization  Statistics  (in  meters) 


Mean 

Std  Deviation 

1  Position  (x,y) 

dx  -  easting  error 

N/A 

N/A 

dv  -  northing  error 

N/A 

N/A 

Radial  error 

N/A 

N/A 

iDepth  (z) _ J 

dz  -  averaged  depth  error 

-0.24 

0.36 

Idzl®  -  absolute  depth  error 

0.42 

Characterization  Statistics 


#  Baseline  #  Correct  Pc° 

Ability  to  Type 

Ordnance 

17 

17 

1.00 

Nonordnance 

3 

0 

0.00 

Ability  to  Size 

Large 

5 

2 

0.40 

Medium  v 

7 

4 

0.57 

Small 

5 

5 

1.00 

Ability  to  Classify 

Bomb 

4 

2 

0.50 

Projectile 

9 

2 

0.22 

Mortar 

4 

0 

0.00 

Submunition 

0 

0 

NA 

Rocket 

0 

0 

NA 

Notes:  *  Square  root  of  the  mean  square  depth  error 

’’  Probability  of  correct  characterization 


5.4.6 -1 


SCA.Geometrics  -  Interrogation/Burial  Area 


Localization  Statistics  (in  meters) 


1  Mean 

Std  Deviation 

Position  (x,y) 

dx  -  easting  error 

-0.06 

0.09 

dy  -  northing  error 

-0.07 

0.11 

Radial  error 

0.15 

0.08 

iDepth  (z) _ 

dz  -  averaged  depth  error 

0.09 

0.5 

Idzl®-  absolute  depth  error 

0.50 

Characterization  Statistics 


#  Baseline  #  Correct  Pc** 

Ordnance 

17 

16 

0.94 

Nonordnance 

3 

0 

0.00 

lAbilitv  to  Size 

5 

2 

0.40 

Medium 

6 

4 

0.67 

Small 

6 

6 

1.00 

Bomb 

4 

3 

0.75 

Projectile 

8 

3 

0.38 

Mortar 

5 

1 

0.20 

Submunition 

0 

0 

NA 

Rocket 

0 

0 

NA 

Notes:  “  Square  root  of  the  mean  square  depth  error 

Probability  of  correct  characterization 
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6.0  RESULTS  SUMMARY,  ANALYSIS,  AND  CONCLUSIONS 


The  objectives  of  the  Phase  HI  ATD  include  the  development  of  relevant  performance 
information  on  UXO  technologies  demonstrated  under  realistic  constraints.  The 
discussions  that  follow  summarize  results  based  on  demonstrator  scenario  performances. 
An  analysis  of  overall  survey  and  demonstrator  excavation  performance  is  also  provided. 


6.1  SCENARIO  STATISTICAL  PERFORMANCE  SUMMARIES 

The  key  technical  issue  in  characterizing  or  remediating  properties  with  UXO  is  the 
effectiveness  of  ordnance  detection.  Given  that  a  target  is  detectable,  target  localization 
will  affect  the  efficiency  of  target  reacquisition  for  subsequent  statistical  sampling  or 
identification  and  remediation.  Ordnance  characterization  (typing)  will  further  affect 
remediation  costs,  as  it  is  a  controlling  parameter  in  the  number  of  UXO  unearthed  per 
day  by  excavators. 

The  clearance  or  sampling  depth  requirements,  ordnance  characteristics,  type  and  quantity 
of  nonordnance  debris,  and  geotechnical/environmental  characteristics  (e.g.  accessibility, 
magnetic  mineral  deposits,  topography,  vegetation,  etc.)  will  affect  the  detection 
performance  of  sensing  technologies.  The  conditions  at  JPG  can  be  described  as 
favorable  for  the  detection  of  ordnance.  The  assessment  of  demonstrator  detection 
performance  in  the  scenarios  emphasizes  the  feasibility  of  detecting  the  baseline  ordnance 
set.  The  false  alarm  metrics  are  a  necessary  consideration  because  of  the  evaluation 
methodology  used  for  determining  detection.  Localization  and  characterization  statistical 
summaries  are  provided  for  all  scenarios.  Demonstrators  who  perform  better  than  chance 
are  assumed  to  have  a  nominal  capability  to  distinguish  ordnance  and  nonordnance. 

GRI’s  multiple  data  sets  and  SC&A’s  teaming  arrangements  are  treated  as  independent 
demonstrations.  SC&A’s  performance  is  occasionally  highlighted  because  of  the 
uniqueness  of  their  approach  in  the  ATD.  SC&A  post  processed  the  raw  data  sets  of 
ADI,  Geo-Centers,  and  Geometries  in  an  effort  to  demonstrate  SC&A’s  unique  detection, 
localization,  and  characterization  capabilities. 
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6.1.1  Scenario  1,  Aerial  Gunnery  Range 


Fourteen  demonstrations  were  conducted  on  Scenario  1.  The  baseline  ordnance  consisted 
of  43  targets,  including  large  bombs  (250  pound  to  750  pound),  rockets  (2.75  inch  and  5 
inch),  and  MK  76  practice  bombs  -  typical  air-launched  ordnance.  Demonstrators  were 
required  to  report  ordnance  targets  to  a  depth  of  3  meters;  however,  no  baseline  ordnance 
targets  were  emplaced  deeper  than  2  meters. 

The  detection  performance  of  demonstrators,  as  grouped  by  their  sensor  technology,  is 
summarized  in  Table  6.1.1-1.  The  detection  probabilities  are  based  on  the  demonstrator 
searching  the  entire  scenario.  It  should  be  noted  that  Foerster’s  probability  of  ordnance 
detection,  based  on  the  area  they  searched,  is  (to  be  provided  when  the  data  is 
available).  Geo-Centers  scored  a  commendable  100  percent  detection  of  the  baseline 
targets  in  this  scenario,  but  at  the  expense  of  a  moderate  false  alarm  rate.  Over  40  percent 
(6  of  14)  of  the  demonstrators  had  Pd  ’s  greater  than  85  percent.  The  success  of 
demonstrators  who  combined  magnetometer  or  gradiometer  with  electromagnetic 
induction  sensors  should  be  noted,  as  they  were  all  above  average  performance.  The 
trend  for  false  alarm  rates  and  ratios  is  not  as  distinguishable  by  technology,  but 
magnetometer  or  gradiometer  combined  with  electromagnetic  induction  were  generally 
below  the  demonstrator  average.  Only  three  of  the  demonstrators  with  Pd’s  of  at  least 
0.60  had  false  alarm  ratios  below  4.  SC&A  processed  ADFs  magnetometer  data  and 
reduced  ADFs  false  alarm  rate  by  ~50  percent,  while  reducing  ADFs  probability  of 
detection  from  79  percent  to  74  percent,  a  ~6  percent  reduction  ratio.  Likewise  SC&A 
reduced  Geo-Centers’  false  alarm  rate  by  ~30  percent  at  the  expense  of  a  7  percent 
reduction  in  ordnance  detection  (100  percent  to  93  percent).  The  relationship  of  the 
probability  of  detection  to  the  false  alarm  metrics  is  shown  in  figure  6.1.1-1  and  6.1.1-2. 
Better  performance  is  in  the  upper  left  hand  comer  of  these  plots,  where  NAEVA  and 
Geophex  lie. 

The  mean  radial  (horizontal)  and  depth  errors  for  all  demonstrators  are  presented  in 
figure  6.1.1-3.  Foerster  had  the  best  positioning  performance  in  this  scenario,  using 
gradiometer  technology  to  locate  26  targets  with  an  average  radial  error  of  only  35 
centimeters.  Foerster  and  Rockwell  were  the  top  performers  in  depth  positioning,  with  an 
approximate  absolute  depth  error  of  0.25  meters.  SC&A  data  processing  did  not  improve 
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ADI’S  or  Geo-Centers’  localization  results.  Most  demonstrators  performed  well  as  there 
should  be  little  difficulty  in  re-locating  the  relatively  large  baseline  ordnance  of  this 
scenario  with  horizontal  position  errors  less  than  one  meter,  and  depth  errors  less  than  0.5 
meters. 

Figure  6.1.1-4  shows  that  no  demonstrator  has  an  ability  to  distinguish  baseline  ordnance 
and  nonordnance  from  each  other,  as  no  demonstrator  performed  better  than  chance. 
Demonstrators  who  tried  to  characterize  nonordnance  targets  as  nonordnance  generally 
mistyped  a  comparable  percentage  of  their  ordnance  targets  as  nonordnance. 

Demonstrators  sizing  performance  is  summarize  in  figure  6.1.1-5,  which  shows  most 
demonstrators  have  a  nominal  capability  to  estimate  the  size  (diameter)  of  the  baseline 
ordnance  in  this  scenario.  The  individual  capabilities  are  dependent  on  the  size  of  the 
ordnance  being  estimated.  Because  the  baseline  ordnance  mix  had  a  good  size 
distribution,  {his  scenario  better  represents  the  sizing  capabilities  of  demonstrators  than 
the  other  scenarios.  Foerster,  Geophex  and  SC«&;A_Geo-Centers  and  GRI  (Mag)  were 
among  the  better  performers,  each  with  an  averaged  probability  of  estimating  the  three 
sizes  correctly  50  percent  or  more. 

In  summary,  demonstrators  performed  well  in  detecting  and  localizing  the  baseline 
ordnance,  and  established  the  feasibility  of  detecting  ordnance  with  this  size  and  depth 
distribution.  All  baseline  ordnance  items  were  detected  .  The  false  alarm  metrics  provide 
only  a  relative  index  of  performance  specific  to  JPG  and  they  may  not  be  achievable  on 
an  actual  range.  No  demonstrator  showed  any  capability  to  distinguish  ordnance  and 
nonordnance,  so  excavation  of  false  targets  would  likely  consume  a  sizable  portion  of 
remediation  resources  where  nonordnance  is  prevalent. 
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TABLE  6.1 .1-1 


DEMONSTRATOR  ORDNANCE  DETECTION  METRICS  BY  SENSOR  TECHNOLOGY 
AERIAL  GUNNERY  RANGE 


6- 


PER  HECTARE) 


Figure  6.1 .1-2 


DEMONSTRATOR  LOCALIZATION  PERFORMANCE  FOR  THE  AERIAL  GUNNERY 

RANGE  (SCENARIO  1) 


Figure  6.1. 1-3 


(SUBlHini)  UOUU3  BONVlSia 


DEMONSTRATORS 


PROBABILITY  OF  CORRECT  SIZE  CLASSIFICATION  FOR  THE  AERIAL  GUNNERY 

RANGE  (SCENARIO  1) 


6.1.2  Scenario  2,  Artillery  and  Mortar  Range 


Sixteen  demonstrations  were  conducted  on  Scenario  2.  The  baseline  ordnance  consisted 
of  67  ordnance  items,  including  mortars  (60nim,  Slmm,  and  4.2  inch),  projectiles  ( 
76mm  to  8  inch)  and  rockets  (66mm  and  3.5  inch).  Demonstrators  were  required  to 
search  to  a  depth  of  1.2  meters,  and  no  baseline  ordnance  was  located  below  this  depth. 

The  detection  performance  of  demonstrators,  as  grouped  by  their  sensor  technology,  is 
summarized  in  Table  6. 1.2-1 .  NAEVA  has  a  remarkable  performance  on  this  site,  97 
percent  ordnance  detection  with  the  low  false  alarm  rate  of  19  false  alarms  per  hectare. 
One-half  of  the  demonstrations  (8  of  16)  had  Pd’s  of  more  than  85  percent. 
Electromagnetic  induction  sensors  combined  with  magnetometers  had  the  most  consistent 
high  detection  performance.  Six  of  the  demonstrators  with  Pd’s  of  at  least  60  percent  had 
false  alarm  ratios  of  less  than  4.  SC&A  reduced  both  ADI’s  and  Geo-Centers’  false  alarm 
rates  by  half,  while  respectively  reducing  detection  by  4  percent  (85  to  83  percent)  and  19 
percent  (93  to  76  percent).  SC&A  increased  Geometric’s  detection  by  ~7  percent  (90  to 
96  percent),  but  at  the  expense  of  a  ~9  percent  increase  in  the  false  alarm  rate.  The 
relationship  of  demonstrator  ordnance  detection  probabilities  and  false  alarm  metrics  is 
shown  in  figures  6.1.2-1  and  6.1.2-2.  Better  performance  is  in  the  upper  left  hand  comer 
of  these  plots  where  NAEVA,  SCA_Geometrics,  and  Geometries  lie. 

The  mean  radial  (horizontal)  and  depth  errors  for  all  demonstrators  is  summarized  in 
figure  6.1.2-3.  Nearly  one-half  of  the  demonstrators  located  the  ordnance  targets  below  a 
mean  radial  error  of  0.5  meters.  Five  demonstrators  determined  ordnance  depth  to  within 
a  mean  of  25  centimeters.  The  precision  in  depth  may  be  due  to  the  relatively  shallow 
(<1.2  meters)  target  base.  Most  other  demonstrators  performed  well  in  location  and  depth 
estimation.  There  should  be  little  difficulty  in  re-locating  UXO  of  the  size  found  in  this 
scenario,  with  horizontal  position  errors  of  less  than  one  meter,  and  depth  errors  less  than 
0.5  meters. 

Demonstrator  typing  performance  is  summarized  in  figure  6.1.2-4.  No  demonstrator 
showed  any  significant  ability  to  distinguish  baseline  ordnance  and  nonordnance  from 
each  other.  Demonstrators  who  tried  to  characterize  nonordnance  targets  as  such 
generally  mistyped  a  greater  percentage  of  their  ordnance  targets.  Demonstrators  who  did 
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not  report  nonordnance  had  reported  a  significant  percentage  of  the  nonordnance  baseline 
targets  relative  to  their  ordnance  detection. 

Demonstrators  sizing  performance  is  summarized  in  figure  6.I.2.4.  Many  demonstrators 
have  a  nominal  capability  to  estimate  the  size  (diameter)  of  the  baseline  ordnance  in  this 
scenario.  The  target  base  consisted  primarily  of  small  and  medium  ordnance.  SC&A 
performed  best  in  that  they  correctly  estimated  the  size  of  the  small  and  medium  ordnance 
more  than  50  percent  of  the  time  with  their  Geo-Centers  data. 

In  summary,  demonstrators  performed  well  in  detecting  and  localizing  the  baseline 
ordnance,  and  established  the  feasibility  of  detecting  ordnance  with  this  size  and  depth 
distribution.  All  baseline  ordnance  targets  were  detected.  The  false  alarm  metrics 
provide  only  a  relative  index  of  performance  specific  to  JPG  and  they  may  not  be 
achievable  on  an  actual  range.  No  demonstrator  showed  any  capability  to  distinguish 
ordnance  an^  nonordnance,  so  excavation  of  false  targets  would  likely  consume  a  sizable 
portion  of  remediation  resources  where  nonordnance  is  prevalent. 
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TABLE  6.1 .2-1 


DEMONSTRATOR  ORDNANCE  DETECTION  METRICS  BY  SENSOR  TECHNOLOGY 
ARTILLERY  AND  MORTAR  RANGE 


Sensor  Tvoe 

Demonstrator 

II 

IlSg 

Electromagnetic 

Induction 

(EM) 

CHEMRAD 

0.43 

10.25 

1.66 

0.87 

117.24 

9.47 

GeoPotential 

0.03 

4.27 

10.00 

Magnetometer 

(Mag) 

ADi 

0.85 

76.88 

6.32 

Battelle 

0.12 

1.71 

1.00 

GRi  IMao) 

0.93 

237.25 

17.92 

Rockwell 

0.21 

27.12 

9.07 

SCA  ADI 

0.82 

35.88 

3.05 

EM  &  Gradiometer 

Geoohex 

0.67 

21.14 

2.20 

\ 

EM  &  Mag 

GRI  (Combined) 

0.90 

258.18 

20.15 

Geo-Centers 

0.93 

80.72 

6.10 

Geometries 

0.90 

38.44 

3.00 

NAEVA 

0.97 

19.01 

1.37 

SCA  Geo-Centers 

0.76 

38.01 

3.49 

SCA  Geometries 

0.96 

41.86 

3.06 

Ground  Penetrating 
Radar  &  EM  &  Grad 

ENSCO 

0.70 

43.56 

i  4.34 
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Probability  of  Detection  (PD)  versus  False  Alarm  Ratio 


Figure  6.1. 2-2 


False  Alarm  Ratio  (FA/Ordnance  detected) 


LOCALIZATION  PERFORMANCE  FOR  THE  ARTILLERY  AND  MORTAR  RANGE 

(SCENARIO  2) 


Figure  6.1. 2-3 


DEMONSTRATOR 


Figure  6.1 .2-5 
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6.1.3  Scenario  3,  Grenade  and  Submunition  Mortar  Range 


Eight  demonstrations  were  conducted  for  Scenario  3.  The  baseline  ordnance  consisted  of 
98  ordnance  items,  primarily  small  sized  M42  and  M32  series  submunitions  and  MKl  18 
Rockeyes.  Demonstrators  were  required  to  search  to  a  depth  of  0.5  meters,  and  no 
baseline  ordnance  was  located  below  0.4  meters. 

The  detection  performance  of  demonstrators,  as  grouped  by  their  sensor  technology,  is 
summarized  in  Table  6.1.3-1.  Geo-Centers,  GRI  (EM)  and  GRI  (Combined)  were  the 
only  demonstrators  with  ordnance  detection  above  85  percent.  The  difference  in  GRI’s 
magnetometer  performance  (47  percent)  and  electromagnetic  induction  performance  (92 
percent)  show  the  value  of  EM  technology  in  detecting  small,  relatively  shallow 
ordnance.  Only  two  demonstrators  (SCA_Geo-Centers  and  Geo-Centers)  had  Pd’s  of  at 
least  60  percent  with  false  alarm  ratios  less  than  or  equal  to  4.  SC&A  reduced  both 
ADFs  and  Geo-Centers’  false  alarm  rates  by  ~63  percent  and  ~53  percent,  but  with 
respective  reductions  of  ~37  percent  (from  72  to  46  percent)  and  ~24  percent  (from  91  to 
69  percent)  in  detection  ratios.  The  relationship  of  demonstrator  ordnance  detection 
probabilities  and  false  alarm  metrics  is  shown  in  figures  6.1.3-.1  and  6.1.3-2.  Better 
performance  is  in  the  upper  left  hand  comer  of  these  plots  where  Geo-Centers  lies. 

The  mean  radial  (horizontal)  and  depth  errors  for  all  demonstrators  is  shown  in  figure 
6.1.3-3  in  this  scenario.  Most  demonstrators  could  localize  the  targets  to  within  0.5  mean 
radial  error,  and  provide  reasonable  depth  estimates  for  the  small  targets.  (Note: 
GeoPotential’s  and  SCA_Geo-Centers’  depth  data  are  not  significant  because  both  their 
statistics  are  based  on  one  depth  reported  for  evaluation).  GRI  (EM)  and  GRI 
(Combined)  has  the  best  performance  in  localizing  the  submunitions  in  this  scenario. 
When  the  precision  of  location  (34  centimeters  or  greater  in  this  scenario)  is  many  times 
the  dimensions  of  a  small  target  such  as  a  M32  submunition  (-  30mm),  then  the  adequacy 
of  location  is  dependent  on  the  site-specific  spatial  density  of  ordnance  and  “noise” 
sources.  Precision  in  position  is  needed  to  the  extent  one  has  confidence  that  target 
excavations  match  detected  anomalies. 

Demonstrator  typing  performance  is  summarized  in  figure  6.1.3-4.  No  demonstrator 
showed  an  ability  to  distinguish  baseline  ordnance  and  nonordnance  from  each  other, 
although  GRI  has  some  of  the  better  performance  seen  in  this  category.  GRI  correctly 
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declared  89  of  90  targets  as  ordnance  (99  percent)  and  correctly  declared  5  of  24  targets 
as  nonordnance  (21  percent). 

Demonstrators  sizing  performance  is  summarized  in  flgure  6.1.3-5.  Most  demonstrators 
performed  well  in  declaring  small  targets  correctly.  However,  demonstrators  were 
informed  to  expect  small  ordnance  on  this  site,  so  the  data  is  probably  biased  in  this 
regard. 

In  summary,  demonstrators  established  the  feasibility  of  detecting  ordnance  of  this  size 
and  depth  distribution.  Only  one  submunition  was  not  detected  by  any  of  the 
demonstrators.  The  false  alarm  metrics  provide  only  a  relative  index  of  performance 
specific  to  JPG  and  they  may  not  be  achievable  on  an  actual  range.  No  demonstrator 
showed  any  significant  capability  to  distinguish  ordnance  and  nonordnance.  The 
excavation  of  false  targets  would  be  a  resource  problem  because  the  small  submunitions 
may  be  diffit^ult  to  distinguish  from  shrapnel  and  metallic  scrap. 
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TABLE  6.1 .3-1 


DEMONSTRATOR  ORDNANCE  DETECTION  METRICS  BY  SENSOR  TECHNOLOGY 


GRENADE  AND  SUBMUNITIONS  RANGE 


Demonstrator 

il 

False  Alarm  Ratio 
f#/Ordnance  Detected) 

0.92 

144.92 

6.68 

GeoPotential 

0.01 

6.51 

27.00 

Maanetometer 

GRI  (Maq) 

0.47 

193.62 

17.46 

EM  &  Mag 

ADI 

0.72 

150.70 

8.80 

0.95 

234.62 

10.46 

Geo-Centers 

0.91 

85.84 

4.00 

SCA  ADI 

EX!^ 

_ 55.94 _ 

_ 5JS _ 

0.69 

¥ 


V 
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PROBABILITY  OF  DETECTION  (Pd)  VERSUS  FALSE  ALARM  RATE  FOR  THE 
GRENADE  AND  SUBMUNITION  RANGE  (SCENARIO  3) 


Figure  6.1. 3-1 


FAR  (#  PER  HECTARE) 


PROBABILITY  OF  DETECTION  VERSUS  FALSE  ALARM  RATIO  FOR  THE  GRENADE 

AND  SUBMUNITION  RANGE  (SCENARIO  3) 


Figure  6.1. 3-2 


LOCALIZATION  PERFORMANCE  FOR  THE  GRENADE  AND  SUBMUNITION  RANGE 

(SCENARIO  3) 


Figure  6.1 .3-3 
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Figure  6.1 .3-4 
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6.1.4  Scenario  4,  Interrogation  and  Burial  Area 


Seven  demonstrators  participated  in  the  Scenario  4  characterization  demonstrations.  The 
targets  they  were  expected  to  classify  included  17  ordnance  items  (4  bombs,  10 
projectiles  and  3  mortars)  and  3  nonordnance  items.  Demonstrators  were  informed  that 
there  were  no  targets  deeper  than  2  meters.  Since  demonstrators  were  provided  the 
geodetic  position  of  the  targets,  no  summary  of  their  localization  statistics  is  provided. 
Some  demonstrators  provided  the  position  data  and  the  error  statistics  are  provided  in  the 
individual  performance  charts  of  Section  5.4.  (Note  in  this  section  that  Geometries 
provided  only  13  target  positions  in  their  target  report  that  matched  their  assigned  20 
target  interrogation  set.  SC&A  used  Geometries  data  and  reported  all  20  positions 
correctly.  Demonstrators  needed  to  provide  a  position  within  two  meters  of  each  marked 
target  for  their  target  report  data  to  be  considered  for  evaluation).  Table  6.1.4-1 
summarizes  the  performance  of  demonstrators  grouped  by  technology  in  estimating  the 
depth  of  ordnWe  targets,  typing  ordnance  and  nonordnance,  and  estimating  the  size  of 
ordnance  targets  and  class.  The  numbers  of  targets  is  this  scenario  is  small,  so  that  the 
data  is  statistically  weak.  The  brief  discussion  that  follows  is  drawn  upon  the  point 
estimates  of  the  performance  data,  primarily  because  this  scenario  allowed  demonstrators 
to  focus  their  resources  on  characterization  of  marked  targets. 

ENSCO,  with  ground  penetrating  radar,  gradiometer,  and  metal  detector  had  the  best 
performance  in  estimating  depth  (0.36  meters),  but  their  performance  in  this  scenario  was 
no  different  from  their  performance  on  the  Aerial  Gunnery  Range  (0.35  meters) .  No 
demonstrator  correctly  typed  any  of  the  3  nonordnance  targets,  thus  again  showing  the 
inability  of  demonstrators  to  distinguish  ordnance  and  nonordnance  baseline  targets. 
Geometries  had  the  best  overall  performance  in  estimating  the  size  and  class  of  the 
ordnance,  but  they  performed  as  well  on  the  Artillery  and  Mortar  Range.  SC&A 
improved  significantly  upon  ADI’s  sizing  performance  for  medium  and  large  targets. 

In  summary,  the  value  received  out  of  this  scenario  was  limited.  No  demonstrations  of 
innovative  target  interrogation  (vice  “search”)  technologies  were  conducted.  The 
characterization  performance  of  the  demonstrators  was  comparable  to  what  was  achieved 
in  the  other  scenarios.  However,  a  “characterization”  range  does  allow  a  variety  of  target 
sizes  and  depths  to  be  emplaced.  This  variety  can  be  used  to  assess  demonstrator 
capabilities  without  the  influence  that  a  scenario  might  have  on  decisions. 
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DEMONSTRATOR  CHARACTERIZATION  CAPABILITIES  BY  SENSOR  TECHNOLOGY 


TABLE  6.1.4  - 1 
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6.2  OVERALL  DEMONSTRATOR  SEARCH  PERFORMANCE 


A  secondary  objective  for  JPG  Phase  III  is  to  continue  the  characterization  of  UXO 
technologies  established  in  the  earlier  phases.  The  analysis  of  overall  demonstrator 
search  performance  is  based  on  the  combined  statistics  (Scenarios  1,  2,  and  3)  of 
Appendix  B,  Demonstrator  Summary  Reports.  The  scenario  numbers  that  demonstrators 
participated  in  are  annotated  to  their  names  in  charts  and  plots.  The  performance  of 
survey  demonstrators  is  based  on  their  abilities  to  detect,  localize  and  characterize  an 
emplaced  set  of  baseline  targets.  The  small  survey  areas  limit  the  value  of  distinguishing 
performance  based  on  the  platform  used  (see  Section  4,  or  Appendix  B  for  the 
demonstrators’  platform  technologies). 

6.2.1  Detection  Performance 

V 

¥ 

The  overall  ordnance  detection  performance  of  demonstrators  at  JPG  Phase  HI,  as 
grouped  by  their  sensor  technology,  is  summarized  in  Table  6.2.1-1.  Three  of  the 
demonstrators  who  had  participated  in  at  least  two  search  scenarios,  Geo-Centers, 
NAEVA,  and  GRI  (Combined),  scored  above  90  percent.  All  three  used  the  combination 
of  sensor  technology  that  was  most  successful  in  Phase  11,  magnetometer  and 
electromagnetic  induction.  NAEVA  is  further  distinguished  because  not  only  did  they 
achieve  the  highest  Pd,  but  also  their  false  alarm  rate,  24.8  false  alarms  per  hectare,  is 
relatively  low  among  all  demonstrators.  Geo-Centers’  false  alarm  rate  is  over  three  times 
that  of  NAEVA’s,  and  GRI  (Combined)’s  is  ten  times  NAEVA’s.  In  examining  the  false 
alarm  ratio,  a  measure  of  the  ordnance  likelihood  in  target  excavations,  over  one-third  of 
NAEVA’s  target  excavations  would  have  resulted  in  ordnance.  This  rate  is  twice  that  of 
Geo-Centers’  16  percent  rate  and  more  than  5  times  GRI’s  6  percent  rate.  (It  should  be 
noted  that  GRI  (Combined)  has  a  high  false  alarm  because  they  elected  to  report  over 
1200  nonordnance  targets  that  were  counted  as  “false  alarms”.  If  GRI  (Combined)  had 
chosen  not  to  report  nonordnance  targets,  they  would  have  reduced  their  false  alarm  rates 
by  almost  40  percent  while  dropping  their  ordnance  detection  by  only  6  percent).  The 
relationship  of  Pd  to  false  alarm  rate  and  false  alarm  ratio  is  shown  for  all  demonstrators 
in  figures  6.2.1-1  and  6.2.1-2.  These  ordnance  detection  statistics  are  a  marked 
improvement  over  Phase  II  where  the  best  performing  demonstrator  (Parsons)  achieved 
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85  percent  ordnance  detection.  Over  40  percent  (6  of  14)  of  the  demonstrations  in  Phase 
En  exceeded  Parson’s  performance  in  Phase  H. 

The  trends  in  detection  performance  and  false  alarm  rates  for  Phases  1, 11  and  HI  are 
shown  in  figure  6.2.1-3.  There  is  a  tendency  for  the  later  Phases  to  move  up  on  the  plot; 
that  is,  the  probability  of  detection  has  been  generally  improving  since  Phase  I.  However, 
there  has  been  no  movement  to  the  left  of  the  chart  that  would  show  false  alarms 
decreasing.  (The  value  of  conducting  trials  at  JPG  is  that  the  performance  trend  in  false 
alarm  rates  can  be  compared  to  a  baseline,  since  false  alarm  rates  are  site-specific). 
Looking  at  just  the  statistics  for  ground-based  demonstrations  that  detected  at  least  50 
percent  of  the  baseline  ordnance,  the  following  trends  are  established  for  the  three  JPG 
Phases: 


TABLfi  6.2.1-2.  Ordnance  Detection  by  JPG  Phase 


#  Ground-Based 
Demonstrators 

Average  Pd 

Average  False 
Alarm  Rate 

Phase  1 

1  of  20 

0.62 

149 

Phase  It 

9  of  12 

0.68 

60 

Phase  III 

14  of  16 

0.77 

77 

The  data  also  shows  the  percentage  of  ground-based  demonstrators  detecting  more  than 
50  percent  of  the  baseline  ordnance  has  increased  since  Phase  I. 


One  issue  is  whether  the  improvements  in  ordnance  detection  seen  in  Phase  III  are  real, 
since  Phase  I  and  11  demonstrators  had  to  look  for  deeper  ordnance.  The  depth  and  size  of 
the  baseline  ordnance  influence  how  well  ordnance  can  be  detected.  There  is  evidence 
that  the  improvement  in  detection  is  not  dependent  on  the  maximum  depth  (2  meters)  of 
the  Phase  HI  baseline  ordnance  set.  The  Phase  n  detection  results  were  recalculated  with 
only  those  ordnance  baseline  targets  that  were  shallower  than  2  meters.  The  recalculated 
results  showed  only  minor  changes  in  the  Phase  II  demonstrator  performance,  and  the 
average  Pd  for  the  9  Phase  II  demonstrators  who  scored  at  least  50  percent,  actually 
deceased  from  68  percent  to  67  percent. 
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The  change  in  the  average  ordnance  depth  from  0.9  meter  in  Phase  II  to  0.4  meter  could 
have  a  significant  bearing  on  the  improvement  in  performance  that  was  seen  in  Phase  HI, 
but  its  influence  is  not  so  easily  determined.  Detection  usually  follows  a  power 
relationship  proportional  to  the  distance  a  sensor  is  from  a  target,  and  that  makes  nearby 
targets  much  easier  to  detect.  For  example,  a  total  field  magnetometer  has  a  cubed 
distance  relationship:  a  magnetometer  0.5  meter  off  the  ground  would  see  a  370  percent 
increase  [(1. 4/0.9)  in  target  strength  for  a  steel  target  at  0.4  meters  depth  than  the  target 

strength  for  the  same  target  at  0.9  meters  depth.  This  average  increase  in  target  strength 
must  be  discounted,  because  while  Phase  HI  ordnance  was  shallower,  it  was  also  smaller 
than  the  Phase  II  ordnance.  Small  targets  represented  75  percent  (155/208)  of  the  Phase 
in  baseline  ordnance  but  only  44  percent  (69/158)  of  the  Phase  II  baseline  ordnance  (on 
the  “16A”  site).  Small  targets  are  more  difficult  to  detect,  and  not  just  because  there  is 
less  mass.  Target  volume  also  affects  detection,  A  55  gallon  oil  drum  has  a  much 
stronger  magnetic  “signature”  than  a  comparable  mass  of  metal  collapsed  into  a  sphere. 
An  addition^^consideration  in  the  Phase  11-111  differences  in  detection  performance  is 
the  dispersal  of  ordnance  in  Phase  III  (no  ordnance  within  2  meters  of  other  ordnance) 
relative  to  Phase  n  (37  percent  of  the  ordnance  to  within  2  meters  of  other  ordnance). 
Demonstrators  could  take  advantage  of  the  target  strength  of  “grouped”  ordnance  in 
Phase  II,  while  Phase  IH  demonstrators  could  not. 

The  general  improvement  in  demonstrator  detection  performance  may  be  due  to  the 
scenario  aspect  of  the  ATD.  Demonstrators  could  apply  their  technology  to  scenarios  that 
were  within  their  capabilities.  Detection  of  ordnance  under  the  conditions  of  the 
scenarios  was  adequately  demonstrated. 
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TABLE  6.2.1-1 


DEMONSTRATOR  ORDNANCE  DETECTION  BY  SENSOR  TECHNOLOGY 
COMBINED  SCENARIOS  (1, 2,  AND/OR  3) 


Sensor  Tvne 

Demonstrator  (Scenario  # ) 

■ 

False  Alarm 
(FA)  Rate 

(#/Hectare) 

FA  Ratio 
(#/Ordnance 

Detected) 

Electromagnetic 
Induction  (EM) 

CHEMRAD  (1.2) 

0.50 

1.91 

GRI  rEMl  n.2.31 

0.87 

123.89 

8.46 

GeoPotential  (L2.3) 

0.06 

8.54 

Foerster(l) 

0.60 

36.46 

4.85 

Magnetometer 

(Mag) 

Battelle  (2) 

0.12 

1.71 

GRI  (Mae)  (1.2.31 

0.70 

223.68 

18.82 

Rockwell  (1.2) 

0.34 

25.93 

5.70 

EM  &  Grad 

Geoohex  (1.2) 

0.77 

32.44 

3.11 

EM  &  Mag 

ADI  (3:ma2  onlv  in  1.2) 

0.78 

109.48 

8.30 

GRI  (Combined)  (1.2.3) 

0.93 

240.53 

15.23 

Geo-Centers  (1.2.3) 

0.93 

81.80 

5.18 

0.90 

38.44 

3.00 

NAEVA(1.2) 

0.94 

24.84 

1.96 

SCA  ADI  (3:mae  onlv  in  1.2) 

0.63 

46.80 

4.36 

SCA  Geo-Centers  (1.2.3) 

0.76 

43.55 

3.36 

SCA  Geometries  (2) 

0.96 

41.86 

3.06 

Ground  Penetrating 
Radar  &  EM  &  Grad 
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PROBABILITY  OF  DETECTION  (Pp)  VERSUS  FALSE  ALARM  RATE  (#FA/HECTARE) 
COMBINED  SCENARIOS  (EXCEPT  SCENARIO  4) _ 


Pd  versus  false  ALARM  RATE  COMPARISON 
(Phase  I,  Phase  II,  Phase  III) 


(aJBjoaii/#)  avj 


6.2.2  Localization  Performance 


The  performance  of  all  demonstrators  is  summarized  in  figure  6.2.2- 1.  GRI  (EM  and 
Combined)  was  one  of  the  better  overall  performers  in  locating  ordnance  in  two  or  more 
survey  scenarios,  as  they  had  a  mean  radial  error  of  39  centimeters.  The  choice  of 
navigation  technology  will  determine  how  well  a  demonstrator  can  precisely  locate 
ordnance  in  the  horizontal  plane. 

Navigation  technology  was  divided  into  two  principal  categories.  Differential  Global 
Positioning  System  (DGPS)  technology  is  capable  of  providing  decimeter  geodetic 
positioning  real  time.  NAVSTAR  satellite  visibility  is  essential;  that  is,  the  DGPS 
antenna  needs  line-of-sight  to  the  horizon  to  take  advantage  of  all  satellites  in  view. 
Obstructions  Such  as  trees  and  tall  buildings  can  cause  problems.  Local  grid  reference 
systems  are  relative  navigation  systems  using  chains,  strings,  or  odometers.  Local  grid 
systems  are  relatively  inexpensive,  effective,  and  reliable,  but  they  can  be  labor  intensive. 
Demonstrators  using  local  grid  reference  technology  generally  took  advantage  of  the  30.5 
meter  grid  already  established  at  JPG. 

The  performance  data  in  Table  6.2.2-1  was  averaged  for  Phase  in  ground-based 
demonstrators  (Battelle  used  an  airborne  platform)  and  shows  that  demonstrators  who 
used  a  local  grid  reference  outperformed  DGPS  users,  0.47  meters  vice  0.69  meters  radial 
error.  The  results  for  both  groups  appear  to  be  satisfactory  for  the  relocation  of  ordnance 
(average:  0.55  meters).  The  radial  error  performance  of  ground-based  demonstrators  has 
improved  since  Phase  I,  as  shown  in  figure  6.2.2-2. 
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Figure  6.2.2-2  Radial  Error  by  JPG  Phase 

Location  Accuracy  of  Ground-Based  Demonstrators 
by  Phase 


1  2  3 

JPG  Phase 


The  mean  radial  error  has  decreased  on  average  by  more  than  50  percent  since  the  initial 
Phase  I,  reducing  the  uncertainty  in  target  location.  However,  the  improvement  in 
location  in  Pli^e  HE  may  also  be  due  to  the  shallower  and  dispersed  baseline  ordnance,  as 
these  targets  may  be  easier  to  fix  in  horizontal  space  than  the  deeper  and  grouped  Phase  I 
and  n  targets. 

The  depth  positioning  of  targets  is  dependent  on  sensor  technology.  The  three  ground- 
based  demonstrators  who  used  only  electromagnetic  induction  sensors  [CHEMRAD, 
GeoPotential,  and  GRI  (EM)],  had  the  poorest  depth  performance.  The  average  depth 
performance  of  all  ground  based  Phase  in  demonstrators  (0.40  meters)  improved 
significantly  upon  the  depth  performance  of  the  Phase  II  ground-based  demonstrators 
(0.82  meters).  Again.,  the  shallower  and  more  dispersed  baseline  ordnance  of  Phase  IH 
may  account  for  the  improvement  seen. 
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DEMONSTRATOR  LOCALIZATION  PERFORMANCE,  COMBINED  SCENARIOS 

(EXCEPT  SCENARIO  4) 


Figure  6.2.2-1 


DEMONSTRATOR 


TABLE  6.2.2-1 


DEMONSTRATOR  POSITIONING  PERFORMANCE  BY  NAVIGATION  TECHNOLOGY, 


COMBINED  SCENARIOS  (1,2,  AND/OR  3) 


- 

Mean  Radial 

Absolute  Depth 

Naviaation  System 

Demonstrator  (Scenario  #> 

Error 

Error 

Battelle  (2^ 

3.37 

1.06 

CHEMRADn.2^ 

1.08 

0-68 _ 

Differential 

Foerster  (1) 

0.35 

0.26 

GPS 

Geo-Centers  (1,2,3) 

0.53 

0.48 

GeoPotentlal  (1.2.3) 

0.73 

0.69 

Rockwell  (1,2) 

0.71 

0.26 

SCA  Geo-Centers  (1.2.3) 

0.74 

0.36 

ADI  (1.2.3) 

0.47 

0.46 

0.50 

0.30 

Local  Grid 

Geometries  (2) 

0.39 

0.32 

Reference 

Geoohex  (1,2) 

0.51 

0.26 

V 

GRI  (1.2.3) 

0.39 

0.41 

V 

GRI  (EM)  (1.2.3) 

0.39 

0.48 

GRI  (Mao)  (1.2.3) 

0.48 

0.46 _ 

NAEVA(1.2) 

0.58 

0.28 

SCA  ADI  (1.2.3) 

0.54  . 

0.36 _ 

SCA  Geometries  (2) 

0.41 

_ 0,26 _ 
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6.2.3  Target  Characterization  Performance 


Characterization  performance  is  determined  by  a  demonstrators  ability  to  type,  size  and 
classify  detected  ordnance. 

6.2.3. 1  Typing  Performance 

The  overall  performance  of  demonstrators  in  Phase  HI  in  typing  targets  as  ordnance  and 
nonordnance  is  shown  in  figure  6.2.3.1-1.  This  figure  shows  the  conditional  probability 
of  typing  targets  correctly.  Data  is  not  shown  for  Rockwell,  Foerster,  CHEMRAD, 
ENSCO,  or  Battelle  because  these  demonstrators  typed  their  ordnance  targets  as 
“unknown”.  In  general,  demonstrators  lack  a  capability  to  distinguish  ordnance  and  the 
emplanted  nonordnance,  as  they  did  in  Phase  I  and  Phase  H.  While  GRI  (combined)was 
able  to  classify  the  highest  percentage  of  emplanted  nonordnance  correctly  at  1 1  percent, 
they  mistyped  a  significant  6  percent  of  their  ordnance  detections  as  nonordnance.  What 
is  not  apparent  from  figure  6.2.3.1-1  is  the  typing  performances  of  Geo-Centers, 
Geometries  and  NAEVA,  who  reported  few  or  no  nonordnance  targets.  It  is  necessary  to 
look  at  their  baseline  nonordnance  detection  performance  to  ascertain  their  capabilities. 

Table  6.2.3. 1-1  Combined  Scenario  Detection  (TMA  Closest) 


Pd,  Ordnance 

Po,  Nonordnance 

Geo-Centers 

0.93 

0.80 

Geometries 

0.90 

0.84 

NAEVA 

0.94 

0.85 

It  is  apparent  in  the  table  that  all  three  demonstrators  reported  an  appreciable  percentage 
of  the  baseline  nonordnance  targets  as  evidenced  by  their  high  PdS  for  nonordnance. 
(Because  the  baseline  nonordnance  target  set  includes  nonmetalic  debris,  nonordnance 
detection  probabilities  should  be  somewhat  smaller  that  ordnance  detection  probabilities). 
If  these  companies  had  a  discrimination  capability  and  had  chosen  not  to  report 
nonordnance  targets,  their  nonordnance  detection  probabilities  should  be  significantly 
lower  than  their  ordnance  detection  probabilities.  Although  there  is  some  evidence  of 
discrimination,  they  do  not  have  a  statistically  significant  capability  to  distinguish 
ordnance  and  nonordnance. 
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The  data  in  Figure  6.2.3.1-2  shows  demonstrators  ranked  success  in  typing  targets  as 
ordnance  for  those  demonstrators  that  attempted  to  do  so.  NAEVA  performed  best  by 
being  correct  37  percent  of  the  time  in  declaring  a  target  as  ordnance.  The  value  of 
SC&A’s  data  processing  is  also  evident  in  that  they  were  able  to  improve  on  ADI’s  and 
GeoCenter’s  abilities  in  determining  targets  as  ordnance. 

Scenario  4,  the  Interrogation/Buriai  Area,  was  established  to  identify  demonstrators  with 
a  discrimination  capability.  There  has  been  an  emphasis  on  trying  to  develop  better 
technology  or  processing  schemes  that  can  recognize  ordnance  as  such  in  situ.  Such 
technology  would  be  advantageous  for  land  characterization  studies,  where  the  goal  may 
be  minimal  statistical  sampling  to  determine  the  extent  of  hazard.  A  sampling  plan  can 
statistically  account  for  a  survey  technology  that  has  an  adequate  probability  of  detection 
and  no  target  discrimination  capability  by  “digging  a  few  more  holes”.  However,  the 
impact  of  degaonstrators’  inability  to  distinguish  ordnance  and  nonordnance  targets  is 
likely  to  be  severe  on  remediation  efforts.  Excavators  will  have  to  commit  significant 
resources  that  will  be  wasted  on  false  alarms.  Better  ordnance  recognition  technology 
would  not  help  if  it  does  not  also  improve  the  rejection  of  false  alarms.  Given  high  false 
alarm  rates,  technology  that  can  better  recognize  detected  targets  as  nonordnance  may  be 
more  economically  useful.  Even  in  a  relatively  benign  environment  like  JPG  (no  shrapnel 
other  than  what  was  emplaced),  there  are  typically  3  to  20  times  more  false  alarms  than 
ordnance. 

An  analysis  of  errors  that  are  acceptable  for  ordnance  and  nonordnance  decisions  would 
require  economic  assumptions  beyond  the  scope  of  this  report.  Such  an  analysis  is 
needed  to  set  realistic  standards  or  guidelines  for  technology  developers  and  users. 
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PROBABILITY  OF  CORRECT  TYPE  CLA 


Ainmveoud 


DEMONSTRATOR 


SUCCESS  AT  DECLARING  TARGETS  AS  ORDNANCE 
(Among  Demonstrators  Who  Tried) 


FIGURE  6.2.3.1-2 


xaaHHOD  aoviNa3H3d 


DEMONSTRATORS 


6.2.3.2  Sizing  Performance 


The  overall  performance  of  demonstrators  in  sizing  the  small,  medium  and  large  ordnance 
is  shown  in  figure  6.2.3.2-1.  Sizing  ability  may  be  useful  in  that  it  provides  a  alternative 
means  to  characterize  ordnance  and  nonordnance  targets.  (For  example,  on  some  sites 
small  targets  could  likely  represent  shrapnel  and  large  targets  ordnance.  Therefore  a 
premium  would  be  placed  on  a  demonstrator  who  could  detect  and  recognize  large  targets 
as  such).  The  combined  statistics  show  that  some  demonstrators  are  better  estimating 
certain  classes  of  ordnance.  However,  these  statistics  are  biased  because  of  the  scenario 
aspect  of  the  demonstration.  That  is,  a  demonstrator  on  Scenario  3  may  logically  declare 
all  targets  small,  regardless  whether  they  have  a  true  ability  to  distinguish  this  target  size 
or  not.  The  results  of  Phase  HI  sizing  performance  are  not  significantly  different  from 
Phase  n.  In  Phase  m,  demonstrators  were  correct  estimating  target  size  54  percent  of  the 
time;  in  Phase  II,  demonstrators  were  correct  56  percent  of  the  time. 
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PROBABILITY  OF  CORRECT  SIZE  CLASSIFICATION  -  COMBINED  SCENARIOS 

(EXCEPT  SCENARIO  4) 


Figure  6.2.3.2-1 


Ainiavaodd 


DEMONSTRATOR 


6.2.3.3  ClassiHcation  Performance 

The  analysis  of  overall  classification  performance,  the  capability  to  distinguish  ordnance 
as  mortar,  bomb,  etc.  is  not  provided,  as  the  merit  of  this  statistic  is  questionable  in  light 
of  the  demonstrators  inability  to  distinguish  ordnance  and  nonordnance. 
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6.3  EXCAVATION  RESULTS  AND  ANALYSIS 


6.3.1  Two  excavation  systems  were  demonstrated  at  JPG  for  Phase  III  of  the  UXO  ATD 
program.  Lockheed  Martin  Advanced  Environmental  Systems  demonstrated  a  Caterpillar 
320L  Low  Ground  Pressure  (LGP)  excavator  with  a  single  remote  operator  control  station 
(OCS).  OAO  Corporation  (OAO)  demonstrated  the  Teleoperated  Ordnance  Disposal 
System  (TODS). 


6.3.2  LOCKHEED  MARTIN  ADVANCED  ENVIRONMENTAL  SYSTEMS 
(LMAES) 

6.3.2.1  The  Caterpillar  320L  Low  Ground  Pressure  excavator  with  remote  operator 
control  station  was  demonstrated  at  the  JPG  16-hectare  demonstration  site  from 
November  20<o  22,  1996.  Lockheed  Martin  was  a  first  time  participant  in  the  UXO  ATD 
program. 

6.3.2.2  Demonstrated  Performance 

LMAES  excavated  1 1  targets  in  the  24  hours  allotted  for  its  demonstration.  Table 

6.3.2.2  provides  details  of  the  demonstration  results.  Several  parameters  are  of  interest 
including  average  travel  rate  to  the  targets  (1.17  kilometers  per  hour),  average  burial 
depth  of  targets  (1.24  meters)  and  average  excavation  time  (0.55  hours  per  hole).  Note 
also  that  most  of  the  targets  were  large  objects. 

It  was  observed  by  on-site  EOD  personnel  that  Lockheed  should  excavate  targets  larger 
than  152  millimeters  in  diameter.  For  the  second  day,  EMI  personnel  made  additional 
target  selections  and  observed  that  Lockheed  was  more  adept  at  excavating  larger  items. 
Results  are  comparable  to  Phase  n  demonstrators  as  shown  in  Table  6.3.5. 
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TABLE  63.2.2 


Lockheed  Excavation  Data 


Target 

Type  of  Target 

Date 

#  of  min. 

Depth 

(meters) 

Excavation 

Time  (hr) 

Comments 

1253 

152mm  projectile 

11/19/96 

N/A 

N/A 

N/A 

■  N/A 

N/A 

Not  found^'^ 

1219 

4.2"  mortar 

11/19/96 

10 

0.5 

1.20 

0.38 

1.17 

Recovered^^^ 

1258 

11/20/96 

97.5 

4.0 

1.46 

1.37 

0.67 

Recovered®^ 

1215 

175mm  projectile 

11/20/96 

56.7 

1.31 

0.35 

Recovered 

1253 

152mm  projectile 

11/20/96 

123.4 

1.23 

0.58 

0.98 

Recovered^*^^^ 

1263 

500-lb  bomb 

11/20/96 

157.5 

16.0 

0.59 

0.98 

0.05 

Recovered 

1272 

250-lb  bomb 

11/20/96 

41.8 

6.0 

0.42 

1.82 

0.43 

Recovered 

1577 

5"  rocket 

11/20/96 

36.6 

4.0 

0.55 

0.42 

0.40 

Recovered 

1269 

SOO^lb  bomb 

11/21/96 

45.7 

0.9 

3.05 

1.44 

0.28 

Recovered 

1229 

500-lb  bomb 

11/21/96 

158.6 

14.0 

0.68 

1.07 

0.45 

Recovered 

231 

500-lb  MK-82 

11/21/96 

25 

3.11 

0.97 

Recovered 

1235 

750-lb  bomb 

11/21/96 

50 

2.0 

1.50 

1.17 

0.30 

Recovered 

1264 

N/A 

11/21/96 

N/A 

N/A 

N/A 

N/A 

N/A 

1266 

N/A 

11/21/96 

N/A 

N/A 

N/A 

N/A 

AVERAGES 

72.98 

5.4 

1.17 

1.24 

0.55 

November  19,  1996,  Lockheed  Martin  Advanced  Environmental  Systems  navigated  to  a  position 
about  75  feet  west 

of  target  #1253.  Lockheed  later  reported  incorrect  coordinate  input  and  was  probably  about  1  longitudinal  degree  in 
error 

the  error  was  later  found  and  corrected.  Target  #1253  was  successfully  excavated  on  November  20,  1996. 

Because  Lockheed  could  not  identify  the  ordnance  item  from  the  command  vehicle,  the  target  was 
visually  located 

using  a  Schoenstadt  metal  detector. 

^^^ockheed’s  staked  positions  at  locations  1264  and  1266  were  within  30  and  2  centimeters  of  the  actual  position 
respectively. 
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6.3.3  OAO  CORPORATION 


6.3.3.1  OAO  demonstrated  TODS  from  November  13  through  17,  1996  at  the  16-hectare 
area  at  JPG.  OAO  was  a  first  time  participant  in  the  UXO  ATD  program. 

6.3.3.2  Demonstrated  Performance 

TODS  excavated  24  targets  over  the  course  of  the  40  hour  demonstration  period.  Table 

6.3.3.2  provides  details  of  the  demonstration  results.  Pertinent  parameters  include 
average  travel  rate  to  the  targets  (1.39  kilometers  per  hour),  average  burial  depth  of 
targets  (0.31  meters)  and  average  excavation  time  (0.38  hours  per  hole).  Note  that  most 
targets  are  spjdl. 
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TABLE  63.3.2 


c 

►AO  Excavation  Data 

Target 

Type  of  Target 

Distance 

to  Target 

(m) 

#of 

min. 

Travel 

Rate 

(km/hr) 

Depth 

(meters) 

Excavation 

Time  (hr) 

Comments 

1664 

MK118(Rockeve) 

30 

2 

0.90 

0.15 

0.28 

Recovered 

1644 

M42  Heat 

62 

2 

1.86 

0.12 

0.22 

Recovered 

1600 

M38  bomblet 

83 

7 

0.71 

0.13 

0.17 

Recovered 

1754 

M42  Heat 

141 

B 

2.12 

0.12 

0.25 

Recovered 

238 

99 

2 

2.97 

1.27 

N/A 

ilHBDil 

1768 

M42  Heat 

79 

8 

Recovered 

1610 

M38  bomblet 

96 

2 

2.88 

0.16 

0.12 

Recovered 

1228 

4.2"  iSortar 

179 

8 

1.34 

1.28 

Recovered 

1239 

60mm  mortar 

25 

1 

1.50 

0.28 

0.32 

Recovered 

1217 

105mm  projectile 

76 

2 

2.28 

0.23 

0.38 

Recovered 

1254 

76mm  projectile 

60 

3 

1.20 

0.17 

Recovered 

314 

30mm  projectile  (4) 

104 

B 

1.56 

0.28 

N/A 

Not  found^^^ 

1196 

76mm  projectile 

243 

B 

3.65 

0.4 

N/A 

Recovered^"^^ 

272 

60mm  mortar 

53 

2 

1.59 

0.08 

N/A 

Not  found*^^ 

1449 

81mm  Illumination 

114 

28 

0.24 

0.12 

N/A 

Recovered 

1441 

60mm  fins 

95 

14 

0.41 

0.17 

N/A 

Recovered 

1172 

60mm  mortar 

135 

8 

1.01 

0.24 

0.90 

Recovered^^^ 

1344 

105  mm  projectile 

(illumination  candle) 

103 

12 

0.52 

0.23 

0.30 

Recovered^"*^ 

1852 

2.75"  rocket 

359 

16 

1.35 

0.65 

0.75 

Recovered 

1289 

251b  practice  bomb 

75 

B 

1.13 

0.41 

0.87 

Recovered 

1282 

251b  practice  bomb 

103 

5 

1.24 

0.35 

0.20 

Recovered 

1288 

251b  practice  bomb 

103 

5 

1.24 

0.27 

0.23 

Recovered 

1284 

25  Ib  practice  bomb 

50 

6 

0.50 

0.33 

0.23 

Recovered 

1850 

2.75"  rocket 

150 

17 

0.53 

0.35 

0.10 

Recovered 

AVERAGES 

109.04 

6.92 

1.39 

0.31 

0.38 
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(1,2.3)  jargets  Nos.  238,  314,  and  272  are  designated  as  "anomalies"  from  Phase  III 
emplacement.  During  emplacement, 

EMI  was  unable  to  locate  these  anomalies.  OAO  unsuccessfully  attempted  to 
locate  these  anomalies. 

As  part  of  their  original  proposal,  OAO  performed  remote  navigation  exercises.  At  these 
target  locations,  OAO  was  given  only  target 

coordinates  for  navigation  purposes.  In  all  three  attempts,  OAO  was  successful  at 
navigating  and  excavating  the  correct  targets. 

N/A  -  not  available 


TODS  performance  in  reducing  excavation  time  is  a  function  of  its  design.  The  system  is 
most  effective  when  excavation  targets  are  (1)  less  than  100  kilograms,  (2)  less  than  1.2 
meters  below  the  ground  surface,  and  (3)  less  than  45  centimeters  in  length. 
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6.3.4  Discussion 


Travel  rate,  target  depth,  excavation  time  and  removal  time  all  factor  into  total  excavation 
time.  Travel  rate  depends  upon  terrain,  foliage,  weather  conditions,  and  obstacles. 

Target  size  and  depth  will  dictate  the  type  of  equipment  used,  in  turn,  affecting 
excavation  time  and  travel  rate  because  excavation  options  are  reduced.  A  large,  deep 
target  can  limit  the  excavation  options  to  a  single  piece  of  equipment. 

Small  shallow  targets  are  sometimes  harder  to  excavate  than  deep,  larger  targets.  For 
example,  the  small  target  gets  lost  in  the  bucket  and  ends  up  in  the  overburden  pile 
because  the  operator  couldn’t  “see”  it  or  “feel”  it  with  his  equipment.  In  turn,  safety  of 
personnel  is  compromised  because  the  overburden  pile  must  be  swept  for  targets.  For 
remote  operated  systems  this  is  especially  true. 

Ultimately,  the  excavation  rate  or  “cleanup  rate”  is  linked  to  accurate  information  about 
the  target.  Is  it  ordnance?  Is  it  2.3  meters  deep  at  this  location?  What  kind  of  ordnance 
item  is  it?  What  is  the  orientation  of  the  ordnance  item  in  the  ground  so  that  excavation 
is  performed  safely.  JPG  provides  a  data  point  to  help  quantify  these  questions. 

6.3.5  Excavation  Demonstration  Conclusions 

Table  6.3.5  shows  comparisons  of  performance  between  Phase  II  and  Phase  III.  An 
attempt  has  been  made  to  compare  excavation  demonstrators  to  each  other  in  a 
quantifiable  way,  however,  the  choice  of  an  excavation  system  depends  upon  size  and 
depth  of  buried  UXO  as  well  as  environmental  conditions.  Realizing  that  there  is  a 
“scenario”  dependency,  there  are  no  significant  changes  in  performance  between  Phases 
n  and  m.  Excavation  tools  that  can  substantially  reduce  time  and  cost  of  remediation 
have  not  appeared. 
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TABLE  6.3.5 


Comparison  of  Phase  II  and  Phase  III  Excavation  Performance  Averages 


Demonstrator: 

Concept 

Engineering 

Wright 

Labs 

Lockheed 

Phase: 

II 

II 

III 

m 

Travel  Rate  (km/hr) 

1.24 

2.83 

1.17 

1.39 

Target  Depth  (m) 

1.00 

1.53 

1.24 

0.31 

Excavation  Time  (hr) 

0.75 

0.57 

0.55 

0.38 
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6.4  CONCLUSIONS  AND  RECOMMENDATIONS 


JPG  Phase  in  focused  on  developing  relevant  performance  data  of  technologies  used  to 
search,  detect,  and  characterize  or  excavate  UXO.  Search  and  target  recognition  has 
never  been  a  trivial  task,  whether  one  is  seeking  Spanish  galleons,  Scud  missiles,  or  anti¬ 
personnel  land  mines.  The  JPG  Phase  III  results  showed  that  state  of  the  art  technology 
exists  that  is  capable  of  detecting  a  substantial  portion  of  the  ordnance  emplaced  for  the 
ATD  scenarios.  In  particular,  the  combination  of  electromagnetic  induction  and 
magnetometer/gradiometer  sensors  proved  to  be  an  effective  combination  in  all  three 
survey  scenarios.  The  top  demonstrators  used  this  sensor  technology  with  different 
platforms  and  different  navigation  systems  to  detect  over  90  percent  of  the  emplaced 
ordnance.  The  ability  ground-based  demonstrators  to  precisely  locate  the  ordnance  was 
also  established  with  an  overall  mean  radial  error  of  0.55  meters.  Demonstrators’ 
abilities  to  size  targets  need  improvement  as  their  size  estimates  were  correct  only  a  little 
more  than  haff  the  time.  The  definitions  or  criteria  for  target  size  may  need  to  be 
reconsidered  so  that  the  capabilities  of  demonstrators  are  better  determined. 

No  demonstrator  proved  even  a  modest  capability  to  discriminate  ordnance  and 
nonordnance,  where  one  could  believe  that  their  declarations  were  correct.  The  problem 
with  discrimination  of  ordnance  and  nonordnance  targets  continues  as  it  has  since  Phase 
I.  This  deficiency  in  technology  may  not  affect  UXO  site  characterization  efforts,  but  it 
should  be  a  major  cost  driver  in  any  UXO  remediation  efforts,  since  excavations  will  be 
slow  and  likely  be  unproductive.  The  demand  for  remediation  of  closed  Department  of 
Defense  properties  and  formerly  used  defense  sites  is  high,  but  the  resources  available  to 
meet  the  demand  are  limited.  Resources  wasted  on  unproductive  excavations  will  limit 
the  amount  of  land  that  gets  remediated  to  the  highest  priority  properties  and  UXO 
situations  (such  as  Spring  Valley  in  Washington  DC). 

The  solution  to  target  discrimination  may  not  be  simple.  There  may  be  no  magnetometer 
technology  that  can  determine  if  a  piece  of  steel  has  explosives  attached  to  it.  However, 
existing  technologies,  including  advanced  data  processing  such  as  SC&A’s  efforts,  need 
to  be  further  developed  so  false  alarms  can  be  reduced,  without  adversely  affecting 
ordnance  detection  performance.  SC&A’s  efforts  are  applauded,  and  their 
recommendation  for  the  establishment  of  standards  for  raw  sensor  data  is  endorsed. 
Furthermore,  standard  sensor  data  sets  need  to  be  publicly  available  to  encourage  the 
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development  of  advanced  processing  algorithms  without  the  expense  and  burden  of  data 
gathering.  Target  discrimination  standards  or  guidance  should  also  be  provided  to 
developers.  How  many  false  alarms,  not  excavated,  are  worth  a  missed  ordnance 
excavation?  Are  any  missed  ordnance  detections  ever  acceptable?  These  are  fair  but 
unanswered  questions  that  discrimination  technology  developers  need  to  consider. 

The  performance  of  two  demonstrators  showed  that  remote  excavation  is  feasible,  but 
their  results  indicate  that  demonstrators  can  find  targets  much  faster  than  they  can  be 
excavated  using  remote  technology.  The  performance  of  excavators  has  not 
substantially  changed  from  Phase  I.  Each  survey  demonstrator  reported  hundreds  of 
targets  in  their  demonstration  periods,  while  the  excavators  only  unearthed  a  few  dozen 
targets.  No  cost  comparison  is  offered  on  the  cost  to  detect  an  UXO  item  vice  the  cost  to 
excavate  it,  but  it  is  apparent  that  the  two  functions  are  being  optimized  in  isolation  from 
the  each  other.  Disregarding  the  false  alarm  issues,  it  may  be  necessary  for  surveyors  to 
consider  efforts  that  would  improve  excavation  productivity,  such  as  centimeter  accuracy 
in  target  depth  positioning. 

JPG  Phase  in  allowed  technologies  to  be  defined  on  the  basis  of  their  strengths. 

However,  current  UXO  technologies  may  be  adequate  to  do  little  more  than  be  able  to 
characterize  the  extent  of  UXO  on  properties  .  The  cost  per  hectare  surveyed  and  the  cost 
per  UXO  remediated  are  figures  of  merit  that  could  be  used  to  define  better  the 
technology  gaps  that  remain.  There  is  a  need  to  continue  the  ATD  efforts  as  it  provides  a 
means  to  separate  the  hyperbole  and  the  performance  as  new  ideas  and  capabilities  are 
developed.  The  JPG  controlled  test  site  is  a  unique  national  resource  for  assessing  UXO 
technologies.  Consideration  should  be  given  to  setting  aside  an  area  for  technology 
developers  to  use  as  they  wish.  Should  additional  demonstrations  be  planned  at  JPG,  the 
following  recommendations  are  offered: 

•  Incorporate  other  UXO  scenarios  as  advisable. 

•  Encourage  system  approaches  to  UXO  detection  and  excavation. 

•  Set  a  consistent  standard  for  reporting  nonordnance  so  that  demonstrators’  false  alarm 
metrics  and  discrimination  (typing)  capabilities  are  better  determined. 

•  Re-examine  the  ordnance  size  standards  to  determine  if  they  should  changed,  or  if 
another  measure,  such  as  target  volume,  might  be  more  useful. 

•  Eliminate  the  need  for  demonstrators  to  classify  ordnance  as  mortar,  bomb,  etc.,  until 
such  time  as  their  ordnance  typing  capabilities  have  developed. 
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APPENDIX  A 


GEOTECHNICAL  AND  HISTORICAL  DESCRIPTION  OF  JEFFERSON 

PROVING  GROUND 

(Excerted  from  the  JPG  Phase  n  Report) 


1.0  TOPOGRAPHIC,  PHYSIOGRAPHIC,  AND  GEOLOGIC  PROPERTIES 

Topographic  relief  in  Jefferson  County  is  influenced  by  the  Ohio  and  Muscatatuck  River 
watersheds.  The  Ohio  River  watershed,  located  in  the  eastern  third  of  Jefferson  County, 
is  very  dissected  and  is  characterized  by  narrow,  sloping  ridges  and  steep  hillsides  with 
terraces.  The  Muscatatuck  River  watershed,  located  in  the  western  two-thirds  of 
Jefferson  County,  in  characterized  by  broad,  nearly  level  ridges  and  moderately  sloping 
hillsides.  The  major  tributary  of  the  Ohio  River  in  Jefferson  County  is  Indian-Kentuck 
Creek,  whichtirains  the  eastern  third  of  the  county  (USDA  1985b). 

Physiographically,  the  demonstration  areas  are  nearly  level  with  a  slightly  undulating 
surface,  marked  by  minor  erosional  features  from  surface  water  runoff.  Both  areas  are 
well-vegetated  with  grasses,  shrubs,  and  trees.  No  tributaries  to  the  Ohio  or  Muscatatuck 
Rivers  dissect  the  demonstration  areas. 

The  demonstration  areas  are  located  on  the  uplands,  in  areas  of  sparse  forestation.  Both 
areas  are  located  adjacent  to  access  roads  along  the  east  side  of  the  facility.  Drainage  at 
the  16A-hectare  area  is  to  the  west  into  Big  Creek.  Drainage  at  the  32-hectare  area  is  to 
the  east  into  West  Fork  Creek. 

Surficial  soils  are  situated  on  a  flat  plain  known  as  the  Dlinoisan  till  plain  (Indiana 
Department  of  Natural  Resources  [IDNR]  no  date).  The  plain  consists  of  glacial  till 
deposited  during  Illinoisan  glaciation.  The  glacial  deposits  are  underlain  by  Silurian-aged 
Laurel  Dolomite  bedrock.  The  Laurel  Dolomite  is  about  14  meters  (45  feet)  thick,  gray, 
and  cherty.  Below  the  bedrock,  Silurian-  and  Ordovician-aged  interbedded  limestone  and 
shale  extends  from  91  to  121  meters  (300  to  400  feet).  Depth  to  bedrock  at  the 
demonstration  areas  ranges  from  1.5  to  9  meters  (5  to  30  feet)  below  ground  surface 
(PRC  1994). 
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Native  soils  at  the  16-hectare  and  the  32-hectare  areas  consist  mainly  of  Avonburg  and 
Cobbsfork  silt  loams.  Avonburg  soils  are  nearly  level,  deep,  and  somewhat  poorly 
drained  soils  situated  on  smooth  uplands.  Areas  of  this  soil  type  are  broad  and  irregular 
in  shape  and  cover  8  to  80  hectares  (20  to  200  acres)  (USDA  1985b).  Cobbsfork  soils  are 
nearly  level,  deep,  and  poorly-drained  soils  situated  on  tabular  divides  in  uplands; 
Cobbsfork  soils  are  prone  to  ponding.  Areas  of  this  soil  type  are  bfoad  and  irregularly 
shaped,  ranging  from  16  to  810  hectares  (40  to  2,000  acres)  in  size  (USDA  1985a). 

Cobbsfork  soils  have  a  very  high  available  water  capacity  and  very  slow  permeability. 
Avonburg  soils  have  a  moderate  available  water  capacity  and  very  slow  permeability.  In 
both  soil  types,  the  water  table  is  typically  perched  at  or  near  the  surface  during  most  of 
the  year.  Both  the  Avonburg  and  Cobbsfork  soils  are  low  in  organic  matter,  and  they  are 
acidic,  friable,  and  best  suited  for  grass  and  tree  development  (USDA  1985a  and  1985b). 


2.0  ECOLOGIC  CHARACTERISTICS 

JPG  consists  primarily  of  poorly  drained  flats  in  various  stages  of  succession  from  open 
fields  to  regrowth  forested  flatwoods.  Flatwoods  are  forested  areas  that  occur  on  level  or 
nearly  level  soils  that  are  poorly  drained  and  have  a  shallow  perched  water  table.  Some 
wooded  stream  valleys  with  better  drainage  are  also  present  at  JPG.  Vegetative 
community  types  that  have  been  inventoried  by  the  IDNR  Division  of  Nature  Preserves 
include  bottomland  forests,  upland  forests,  and  cliffs  along  these  major  drainages  (IDNR 
no  date). 

JPG  lies  within  the  Bluegrass  Natural  Region,  as  identified  by  IDNR.  This  natural  region 
is  identified  and  named  for  its  similarities  in  physiography  and  natural  communities  to  the 
Bluegrass  Region  of  Kentucky.  Most  of  the  natural  region  was  originally  forested, 
although  a  few  glade,  cliff,  and  barren  remnants  are  known,  as  well  as  nonforested 
aquatic  communities.  The  areas  used  for  the  UXO  demonstrations  can  be  classified  as 
Bluegrass  Till  Plain  Flatwoods.  These  natural  communities  are  forested  areas  on  level  or 
nearly  level  soils  that  are  poorly  drained  and  acidic,  with  a  shallow  perched  water  table 
(IDNR  no  date). 

3.0  CLIMATIC  PATTERNS 


Climate  in  Jefferson,  Ripley,  and  Jennings  Counties  is  cold  in  winter  and  hot  in  summer. 
Winter  precipitation  consists  mainly  of  snow,  which  aids  in  soil  moisture  accumulation 
and  minimizes  drought  conditions  in  summer  months.  In  winter,  the  average  temperature 
is  about  2  C  (35  F);  the  average  daily  minimum  temperature  is  about  -4  C  (25  F).  In 
sununer,  the  average  temperature  is  about  24  C  (75  F);  the  average  daily  maximum 
temperature  is  about  30  C  (85  F). 
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The  total  annual  precipitation  is  about  107  centimeters  (42  inches),  with  about  55 
centimeters  (22  inches)  falling  from  April  through  September.  Thunderstorms  occur 
about  50  days  per  year;  tornados  and  severe  weather  also  occur  occasionally.  These  types 
of  storms  are  usually  local  and  short  in  duration  and  can  cause  severe  damage  locally 
(USD  A  1985a,  1985b). 

The  average  seasonal  snowfall  is  about  33  centimeters  (13  inches).  The  average  relative 
humidity  in  midaftemoon  is  about  60  percent.  Humidity  is  higher  at  night,  and  the 
average  at  dawn  is  about  80  percent.  The  sun  shines  70  percent  of  the  time  in  summer 
and  40  percent  in  winter.  Prevailing  winds  are  from  the  south.  Average  wind  speed  is 
highest  in  spring  at  16  kilometers  (10  miles)  per  hour  (USDA  1985a,  1985b). 


4.0  HISTORIC  SITE  USE 

An  extensive  survey  of  historical  data  related  to  the  site  indicates  that  farming  was  the 
predominant  land  use.  The  land  was  typified  by  relatively  small,  dispersed  farmsteads 
and  communities.  Both  woodland  and  agricultural  tracts  occurred  in  the  two  controlled 
site  areas.  In  1 940,  the  federal  government  acquired  the  land;  the  first  round  of 
ammunition  was  tested  at  JPG  on  May  10, 1941  (USAEC  1995). 

As  part  of  the  background  investigation  for  the  Phase  I UXO  ATD  program,  an 
archaeological  investigation  was  performed  in  November  1993.  This  study  revealed  that 
both  areas  were  used  for  agricultural  purposes  before  the  federal  government  acquired  the 
land.  One  site  identified  at  the  32-hectare  area  was  believed  to  be  a  historic  farmstead 
that  was  abandoned  in  1941.  The  farmhouse  was  moved  from  the  JPG  property  to  the 
east,  along  Highway  421.  Two  other  sites  identified  in  the  study  were  of  indeterminate 
historic  affiliation.  None  of  the  sites  identified  were  eligible  for  listing  on  the  National 
Register  of  Historic  Places  (Anslinger  1993). 

Geophysical  and  geotechnical  surveys  were  conducted  in  1994  to  establish  area 
conditions  and  identify  hazardous  conditions.  These  surveys  were  conducted  as  part  of 
the  preparation  for  Phase  I  of  the  UXO  ATD  program.  The  survey  results  identified  no 
hazardous  conditions  to  preclude  the  use  of  these  areas  as  controlled  demonstration  areas. 
However,  given  the  nature  and  mission  of  JPG,  a  considerable  amount  of  the  total  base 
area  has  undoubtably  been  affected  by  munition  testing  and  related  activities. 

Until  September  1995,  JPG  served  as  a  munitions  testing  facility  of  the  Test  and 
Evaluation  Command,  U.S.  Army  Material  Development  and  Readiness  Command. 
During  the  period  of  operation,  JPG’s  mission  was  to  check,  investigate,  and  evaluate 
various  test  items  to  determine  whether  they  conformed  to  specifications  (JPG  1980). 
Between  1942  and  1995,  JPG  conducted  a  variety  of  munitions  tests  throughout  the  base. 
Although  neither  of  the  areas  used  for  the  controlled  site  are  specifically  located  where 
these  tests  took  place,  they  may  be  within  the  “fan”  area  of  several  of  the  impact  fields. 
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The  controlled  site  areas  are  believed  to  have  been  only  minimally  affected  by  historical 
activities  conducted  at  JPG. 


APPENDIX  B 


DEMONSTRATOR  SUMMARY  REPORTS  AND  COMBINED  RESULTS 

B.l  ADI 
B.2  Battelle 
B.3  CHEMRAD 
B.4  ENSCO 
B.5  Foerster 
B.6  Geo-Centers 
B.7  Geometries 
B.8  Geophex 
B.9  GeoPotential 
B.IO  GRI 

B.ll  Lockheed  Martin 
B.l 2  JTAEVA 
B.l 3  OAO 
B.14  Rockwell 

B.l 5  Sanford  Cohen  and  Associates 


1.0  ADI^^ 

ADI  demonstrated  from  October  15  through  17,  1996  at  the  16-hectare  area  at  JPG.  ADI 
also  participated  in  Phases  I  and  II  of  the  UXO  ATD  program. 

1.1  TECHNOLOGY  DESCRIPTION 

1.1.1  Sensor  System  and  Transport  Mode 

1.1.1.1  Magnetometer  System 

The  TM-4  is  a  complete  data  acquisition,  processing,  interpretation  and  documentation 
system  designed  to  detect  and  position  ferrous  contamination  sources.  The  operating 
software  not  only  controls  and  monitors  the  data  acquisition  but  also  generates  an  audio 
tone  at  selected  frequency  ranges  and  filters  interference  from  mains'  electricity.  Field  notes 
can  also  be  recorded  within  the  data.  Magnetic  field  profiles  can  be  viewed  on  the  console 
during  the  survey. 

The  magnetometer  is  capable  of  reading  the  total  magnetic  field  from  up  to  four  sensors  to  a 
sensitivity  of  0.01  nT.  at  a  rate  of  100  times  per  second.  Data  is  collected  along  the  line  at 
predetermined  intervals  and  stored  in  solid  state  memory  for  later  transfer  to  the  processing 
computer. 
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A  proton  precession  type  magnetometer  was  used  as  a  magnetic  base-station.  Data  were 
recorded  at  5  second  intervals  (synchronised  with  the  TM-4)  to  a  resolution  of  0. 1  nT. 

1.1.1.2  EM-61  Time  Domain  EM  System 

The  EM-61  is  a  high  powered,  extremely  sensitive  Time-Domain  Electromagnetic  System 
manufactured  by  Geonics  Limited  of  Mississauga,  Ontario,  Canada. 

The  EM-61  can  collect  data  at  one  time  window  after  the  transmission  coil  is  turned  off. 
The  ADI  EM-61  was  modified  to  provide  an  earlier  time  window  than  the  standard  system 

The  two  receiver  cod  configuration  of  the  EM-61  enables  the  suppression  of  near  surface 
metallic  targets  that  can  mask  or  complicate  the  detection  of  deeper  targets.  This 
suppression  is  accomplished  by  taking  the  difference  between  the  responses  of  the  two 
receiver  coils. 


1.1.2  Recommended  Applications  and  Technology  Limitations 

1.1.2.1  Recommended  Applications 

Magnetometer 

The  TM-4  is  ideally  suited  for  the  detection  of  ferrous  items  in  all  terrains  that  are 
accessible  on  foot.  Multiple  sensor  configurations  are  available  for  areas  in  which 
the  vegetation  density  does  not  restrict  sensor  frame  access.  Large  open  areas  can 
be  surveyed  by  a  vehicle  and  trailer  mounted  system  with  position  recorded  by 
GPS. 
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Electro-magnetic 

The  EM-61  is  suited  for  the  detection  of  both  ferrous  and  non  ferrous  items  in 
fairly  open  areas.  It  can  be  operated  in  close  proximity  to  electrical  power  lines, 
electrified  train  tracks  and  metallic  structures. 

1.1.2.2  Technology  Limitations 

Magnetometer 

The  TM-4’s  main  limitations  are  that  it  should  not  be  operated  in  areas  close  to 
large  metallic  stmctures  (buildings,  fences,  etc.)  or  in  proximity  to  electrified  train 
tracks. 

Magnetic  soils  can  also  be  a  problem  in  that  they  create  noise  that  limits  the 
minimum  size  of  detectable  items. 

Electro-magnetic 

The  EM-61  has  several  shortcomings.  The  size  and  weight  of  the  coils  restrict  its 
operation  to  a  towed  or  two  person  carried  system  that  is  suited  to  open  flat  terrain. 
Horizontal  operation  of  the  towed  instrument  cannot  be  achieved  in  hummocky 
terrain.  The  EM-61  has  only  one  time  window  for  data  collection.  This  window  is 
optimum  for  items  of  a  specific  depth/size.  Certain  items  are  therefore  not 
adequately  sampled. 

1.13  Logistic  Requirements 

1.1.3.1  Magnetometer 

The  portable  TM-4  magnetometer  comes  in  two  carrying  cases,  one  weighing  43 
kilograms  and  the  other  weighing  20  kilograms.  These  cases  are  transportable  as 
personal  baggage  on  airlines  and  are  easily  carried  to  the  site  in  a  sedan. 

At  JPG,  the  survey  was  conducted  by  two,  two  person  teams.  Within  each  team, 
one  person  carries  a  frame  on  which  were  attached  the  sensors  (weighing  2 
kilograms  each)  and  the  positioning  odometer,  while  second  operator  cameds  the 
data  acquisition  equipment  and  battery  pack  (totalling  10  kilograms  in  weight)  the 
two  sets  of  components  being  connected  via  a  5  meter  long  cable. 

Ideally,  the  site  should  be  surveyed  and  marked  with  non  metallic  grid  pegs 
positioned  every  100  metres  or  less.  Control  lines  should  be  positioned 
perpendicular  to  the  survey  direction.  Bright  orange  traffic  cones  used  for  heading 
positioning  should  be  visible  from  the  previous  control  line. 

Data  downloading  can  easily  be  conducted  to  a  laptop  PC  in  the  field.  Data 
processing  and  interpretation  are  also  carried  out  on  the  same  PC. 
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1.1.3.2  Electro-magnetic 


The  EM-61  is  shipped  in  two  aluminium  canying  cases.  One  e«bie-case  weighing 
55  kilograms,  the  second,  45  kilograms.  These  cases  are  usually  shipped  via  a 
commercial  carrier.  The  equipment  can  be  transported  on  site  in  a  van  or  small 
pickup  truck. 

Grid  surveying,  power  and  processing  requirements  are  the  same  as  for  the  TM-4 
magnetometer.  The  ground  surface,  however,  should  be  relatively  smooth,  with 
grasses  and  small  bushes  mowed  to  a  height  of  30  cm  or  less.  At  JPG,  the  EM-61 
coils  were  mounted  on  wheels  and  towed  by  one  operator. 

1.1.4  Data  Acquisition 

1.1.4.1  Grid  Co-ordinates 

In  each  of  the  two  areas  surveyed,  data  were  acquired  in  local  grid  co-ordinates. 
Positions  were  measured  along  survey  traverses  using  an  integrated  odometer 
system  for  the  TM-4  and  wheel  odometer  for  the  EM-61.  Survey  pegs  provided; 
were  ^ised  to  define  control  points  from  which  continuous  odometer  calibrations 
were  performed. 

1.1.4.2  Equipment  Configuration  and  Survey  Specification 

Both  the  TM-4  and  EM-61  were  used  over  all  or  part  of  JPG  Phase  III.  Table  1 
describes  the  equipment  configurations  and  survey  specifications  for  each  Scenario. 

1.1.5  Data  Processing  and  Interpretation 
1.1.5.1  Magnetic  Data  Processing 

The  position  of  the  TM-4  data  were  corrected  using  the  control  line  information. 
The  temporal  magnetic  disturbances  recorded  on  the  base  station  magnetometer 
were  reviewed  to  confirm  that  there  have  been  no  large  variations  in  the  field  that 
would  have  an  impact  on  the  data  collection.  These  data  were  then  stored  as  a  raw 
positioned  data  file  in  an  XYZ  format. 

The  data  were  then  heading  corrected  for  sensor  orientation  and  high  pass  filtered 
to  remove  interference  from  geological  sources  below  10  meters. 

The  raw,  positioned  data  collected  in  a  local  co-ordinate  system  was  converted  to 
the  UTM  Zone  16  co-ordinate  system  as  required  by  the  client.  This  data  is 
supplied  to  the  client  in  digital  format  on  CD-ROM  as  requested. 

The  data  was  then  post  processed  using  commercially  available  and  proprietary 
software  in  order  to  determine  item  location,  depth  and  mass.  Further  processing 
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using  a  proprietary  ordnance  library  enabled  these  items  to  be  classified  according 
to  type,  class,  size  and  azimuth  and  declination  of  the  item  roll  axis. 

1. 1.5.2  Electromagnetic  Data  Processing 

The  position  of  the  EM-61  data  was  corrected  by  using  the  control  line 
information.  These  data  were  then  stored  as  a  raw  positioned  data  file.  The  raw, 
positioned  data  collected  in  a  local  co-ordinate  system  were  converted  to  the  UTM 
Zone  16  co-ordinate  system  as  required  by  the  client  and  supplied  in  the  digital 
format  requested  on  CD-ROM. 

The  top  and  bottom  coil  data  were  then  leveled  in  order  to  minimise  instrament 
and  temporal  variations.  Both  data  sets  were  post  processed  using  both  proprietary 
and  commercial  software  in  order  to  determine  the  location  and  depth  of  the  items 
detected. 

1.1.6  Data  Interpretation 

Data  interpretation  consisted  of  integrating  both  the  EM  and  magnetic  outputs. 
For  fach  target  identified,  position,  depth,  weight,  size,  type  class,  confidence, 
azimuth,  and  declination  was  interpreted  and  provided  in  the  Target  Database 
along  with  comments  relevent  to  the  targets. 

1.1.7  Quality  Assurance 

The  measures  taken  to  ensure  the  credibility  and  quality  of  -both  magnetic  and  EM 
data  include  the  following: 

Field  acquisition  and  data  processing  procedures  were  undertaken  under  ADFs 
accredited  ISO  9001  (License  No:  5696)  work  practices. 

These  work  practices  include: 

•  Monitoring  of  the  data  during  collection; 

•  Keeping  non-target  metallic  objects  far  from  the  sensors; 

•  Viewing  raw  data  during  the  survey  and  prior  to  processing. 

1.2  DEMONSTRATION  RESULTS 

1.2.1  Assumptions 

It  has  been  assumed  in  the  results  that  all  dipoles  recorded,  above  a  2  nT. 
threshold,  on  the  magnetics  dataset,  are  caused  by  ordnance  or  non-ordnance  items. 
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1.2.2  Problems  Encountered 
1.2.2.1  Grid  Surveying 

ADI  encountered  discrepancies  with  the  surveyed  grid,  in  particular  the  south  and 
west  of  the  grid  where  there  were  errors  of  up  to  2  meters  in  the  positions  of  the 
pegs.  A  locd  grid  had  been  made  using  line  “O”  as  the  base  line  for  Scenario  3. 
This  line  was  out  by  approximately  1.5  meters  at  the  point  ’08’  and  0.5  meters  out 
at ’015’. 

Sites  set  up  for  demonstrations  in  which  results  are  judged  for  positional  accuracy 
should  have  properly  surveyed  grids.  Errors  in  grid  peg  positions  are  not 
acceptable.  These  grid  errors  should  be  taken  into  account  during  the 
determination  of  the  results. 
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1.2.3  Description  of  Configuration,  Survey  Specifications  and  Results 


TABLE  1  -  Description  of  Configuration,  Survey  Specifications  and  Results 


Description 

Scenario  1 

Scenario  2 

Scenario  3 

Scenario  4 

Magnetics 

Total  Area  Surveyed 

3.46  hectares 

4.68  hectares 

4.15  hectares 

0.31  hectares 

Sensor  Top 

Elevation  Bottom 

0.8  metres 

0.2  metres 

N/A 

0.2  metres 

N/A 

0.2  metres 

0.8  metres 
0.2  metres 

Line  Spacing 

0.5  metres 

0.25  metres 

0.25  metres 

0.5  metres 

Total  Data  Points 

0.794  million 

1.85  million 

1.85  million 

0.12  million 

Measurement  Resolution 

0.01  nT. 

0.01  nT. 

0.01  nT. 

0.01  nT. 

System  Noise 

Electro-magnetics 

Total  Area  Surveyed 

Sensor  Top  Coil 

0.2  nT. 

Not 

Undertaken 

0.2  nT. 

Not 

Undertaken 

0.2  nT. 

4.35  hectares 

0.45  metres 

0.2  nT. 

0.31  hectares 

0.45  metres 

Elevation  Bottom  Coil 

N/A 

N/A 

0.25  metres 

0.25  metres 

Line  Spacing 

N/A. 

N/A 

0.5  metres 

1.0  metres 

Total  Data  Points 

N/A 

N/A 

0.415  million 

3,000 

Measurement  Resolution 

N/A 

N/A 

0.1  mV. 

0.1  mV. 

System  Noise 

N/A 

N/A 

2.0  mV. 

2.0  mV. 

Total  Ferrous  Interpretations 

395 

420 

395 

20 

Number  Exceeding  75  kg 

22 

11 

2 

2 

Number  between  10  &  75  kg 

42 

52 

14 

6 

Number  below  10  kg 

331 

357 

379 

12 

Total  non-ferrous  items 
interpreted 

N/A 

N/A 

308 

N/A 
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1.2.4  DIGITAL  DATA 


ADI  has  provided  the  following  digital  data  to  PRC  Environmental  Management, 
Inc.,  as  set  out  in  the  Request  for  Quotation: 

Raw  Data 

ADI  have  provided  the  raw  data  in  an  XYZ  format  with  line  numbers  included. 
Each  line  is  broken  at  the  control  lines  used  in  the  survey  and  a  new  line  number  is 
used  for  the  next  portion  of  the  line.  This  data  is  located  with  the  origin  of  the 
block  being  the  south  western  comer  of  the  block. 

1.2.4. 1  Processed  Data 

ADI  has  provided  a  copy  of  the  processed  gridded  data  for  each  of  the  four 
scenarios  in  a  TIFF  Image  format. 

1.2.5  Conclusions 

\ 

•  A  total  number  of  1,230  ferrous  targets  were  detected  and 
interpreted  by  interactive  computer-aided  modelling.  This  analysis 
provided  a  measure  of  position,  depth,  mass,  size  and  orientation 
for  each  target.  A  database  in  the  client’s  required  format  was 
provided  containing  this  information. 

•  The  raw  magnetic  data  (approximately  4.61  million  positioned 
measurements)  were  provided  to  the  client  in  digital  format. 

•  A  total  of  approx.  13  hectares  was  mapped  with  the  two  TM-4  crews 
at  0.25  and  0.5  metres  line  spacing  and  a  total  of  approx.  4.5  hectares 
was  mapped  at  0.5  meter  line  spacing  using  two  EM-61  detectors,  all 
within  the  permitted  40  hour  period. 

•  A  total  of  308  targets  interpreted  as  having  a  non-ferrous  metal 
source  were  detected  and  interpreted  into  the  combined  TM-4  /  EM- 
61  target  database. 


1.2.6.  REFERENCES 

The  references  cited  below  were  used  in  the  undertaking  of  the  survey. 

ADI  Limited,  1994,  Work  Instructions  for  TM-4  Imaging  magnetometer  Data 
Acquisition. 
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ADI  Limited,  1994,  Work  Instruction  for  Data  Processing  and  Interpretation 
of  Imaging  Magnetometer  Data. 

ADI  Limited,  1995,  Work  Instructions  for  EM-61  Deep  Metal  Detector  Data 
Acquisition. 

ADI  Limited,  1995,  Work  Instruction  for  Data  Processing  and  Interpretation 
of  EM-61  Deep  Metal  Detector  Data. 
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ADI  -  Combined  Statistics:  Scenarios  1,2,3 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected  Pd°  P random 

Ordnance 

208 

162  0.78  1  0.143 

Nonordnance 

166 

133  0.80  1 

Total 

374 

295 

Number  False  Alarms 

1345 

False  Alarm  Rate  (#/Hectare) 

109.48 

False  Alarm  Ratio  (#/Ord.) 

8.30 

Probability  False  Alarms 

0.138 

Localization  Statistics  (TMA  Closest)  (In  meters) 


Mean 

Std  Deviation  | 

dx  -  eastinq  error 

-0.01 

0.36 

dy  -  northing  error 

0.16 

0.44 

Radial  error 

0.47 

0.37 

dz  -  averaged  depth  error 

0.26 

0.38  1 

ldzl°-  absolilte  depth  error 

0.46 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Po“ 

#  Correct  (  Pc" 

Ability  to  Type  1 

Ordnance 

208 

162 

0.78 

127 

0.78 

Nonordnance® 

167 

133 

0.80 

5 

0.04 

Ability  to  Size  1 

Larqe 

14 

14 

1.00 

13 

0.93 

Medium 

39 

35 

0.90 

15 

0.43 

Small 

155 

113 

0.73 

71 

0.63 

Ability  to  Classify  1 

Bomb 

21 

21 

1.00 

13 

0.62 

Projectile 

42 

39 

0.93 

0 

0.00 

Mortar 

26 

19 

0.73 

7 

0.37 

Submunition 

97 

70 

0.72 

64 

Rocket 

22 

13 

0.59 

2 

■ESSi 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
^MA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 
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Depth  (meters) 


JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator:  ADI 

Small  Medium  Large 


Principal  Diameter  (millimeters) 


■  Target  Detected 
a  Target  Not  Detected 


2.0  BATTELLE,  PACIFIC  NORTHWEST  LABORATORIES^^ 

The  Sikorsky  Cypher  Unmanned  Aerial  Vehicle  (UAV)  was  demonstrated  as  a  platform 
for  UXO  detection  at  the  JPG  16-hectare  demonstration  site  from  October  29  through 
November  4, 1996.  Battelle  PNL  was  a  first  time  participant  in  the  UXO  ATD  program. 


2.1  TECHNOLOGY  DESCRIPTION 

The  system  that  was  demonstrated  consisted  of  the  Cypher  UAV  with  three  total-field 
magnetometers  mounted  on  it,  a  GPS-based  navigation  system,  a  telemetry  subsystem  for 
aircraft  control  and  data  transmission,  and  a  control  vehicle  (base  station). 


Battelle,  Pacific  Northwest  Laboratories,  UXO  Detection  at  the  Jefferson  Proving 
Ground  Using  the  Sikorsky  Cypher  UAV,  January  1997 
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2.1.1 


Aircraft 


The  Cypher  is  a  unique,  doughnut-shaped,  rotary- wing,  unmanned  aircraft  (Figure  1-  not 
shown).  It  is  approximately  6.5  ft  in  diameter  and  weighs  approximately  235  lb.  The 
sensors  and  associated  components  that  were  installed  on  the  aircraft  for  this  survey 
weighed  22.6  lbs.  The  current  aircraft  is  a  technology  demonstrator  which  does  not  yet 
exhibit  the  performance  characteristics  that  will  be  provided  by  later  operational  versions 
of  the  aircraft.  Nevertheless,  the  performance  features  and  mechanical  characteristics 
that  have  been  built  into  the  current  aircraft  make  it  capable  of  performing  low-altitude 
surveys  with  geophysical  sensors  such  as  magnetometers,  ground-penetrating  radar,  and 
electromagnetic  induction  devices.  These  characteristics  include  the  following: 

•  It  performs  vertical  takeoffs  and  landings. 

•  It  can  hover  or  fly  horizontally  or  vertically. 

•  It  uses  two  coaxial  counter-rotating  blades  that  are  shrouded  to  protect  ground 
personnel. 

•  Computer  control  permits  automated  surveys  and  provides  a  high  degree  of  stability. 


2.1.2  Magnetometers 

Three  Geometries  G822A  total-field  cesium  magnetometers  were  mounted  on  short 
booms  that  extended  radially  outward  from  the  aircraft  as  shown  in  Figure  1 .  These 
measured  the  magnetic  field  simultaneously  at  a  rate  of  20  samples/s  (each  sensor).  The 
magnetometers  were  located  at  angles  of  106 , 180  ,  and  257  (clockwise  looking  down) 
relative  to  a  point  defined  as  the  front  of  the  aircraft.  The  spacing  between  the  right  and 
left  magnetometers  was  3.0  m.  The  magnetic  data,  together  with  aircraft  attitude 
information,  aircraft  coordinates,  and  other  flight  parameters,  were  transmitted  to  the  base 
station  where  they  were  recorded  on  the  hard  disk  of  a  portable  computer  (data 
acquisition  and  display  unit). 

2.1.4  Navigation 

Two  GPS  receivers  were  installed  on  the  Cypher.  The  first  was  a  Trimble  unit  that  was 
an  integral  part  of  the  Cypher’s  Integrated  Flight  Management  Unit  (IFMU).  The  second, 
a  NovAtel  RT2  dual-frequency  unit,  was  installed  specifically  for  this  demonstration  and 
provided  superior  accuracy  (approximately  2  cm).  The  aircraft  position  coordinates 
recorded  during  the  survey  were  provided  by  the  NovAtel  unit.  A  matching  RT2  receiver 
was  mounted  at  known  locations  to  provide  differential  corrections.  The  reference 
locations  used  during  the  survey  were  Monuments  1  and  2  and  stake  0-1. 

2.1.4  Flight  Control 

The  aircraft  is  designed  to  be  controlled  by  digital  signals  produced  by  its  IFMU  and  by 
telemetered  signals  provided  by  a  flight  control  computer  in  the  base  station  vehicle.  The 
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latter  controls  the  flight  path  and  altitude  and  provides  an  automated  (programmed) 
survey  mode.  The  height  of  the  aircraft  above  the  ground  is  measured  by  a  radar  altimeter 
mounted  on  the  bottom  of  the  fuselage.  Fiber-optic  gyros  provide  pitch,  roll,  and  heading 
data. 

2.1.5  Data  Recording 

All  of  the  data  telemetered  from  the  aircraft  during  flight  is  recorded  by  Sikorsky’s  data 
acquisition  system.  This  includes  a  large  amount  of  data  that  is  unrelated  to  the 
geophysical  measurements.  The  magnetic  data,  GPS  coordinates,  GPS  time,  and  aircraft 
attitude  data  are  recorded  separately  by  Battelle’s  data  acquisition  computer.  This  unit 
also  displays  the  sensor  data  and  the  track  of  the  aircraft  in  real  time  on  a  color  monitor. 
The  ability  to  observe  the  data  as  it  is  being  recorded  permits  the  operator  to  periodically 
confirm  that  the  system  is  operating  properly.  The  data  files  are  stored  on  a  removable 
hard  disk  for  later  processing  in  the  laboratory. 

2.1.6  Data  Analysis 

The  magnetic  components  of  the  Cypher’s  engine,  gearbox,  and  electrical  generator 
produce  locaT  induced  magnetic  field  variations  at  each  magnetometer  as  the  aircraft 
pitches,  rolls,  and  changes  heading.  The  errors  associated  with  these  attitude-related 
variations  are  large  compared  to  UXO  signatures  and  must  be  removed.  This  was  done 
by  constructing  a  3-dimensional  lookup  table  of  magnetic  correction  terms  for  each 
magnetometer.  These  provided  the  necessary  corrections  for  each  combination  of  pitch, 
roll,  and  heading  that  occurred  during  the  survey.  High-frequency  magnetic  noise 
produced  by  the  Cypher’s  electrical  generator  and  ignition  system  was  effectively 
removed  by  a  low-pass  Fourier  filter.  The  pitch,  roll,  and  heading  angles  were  used 
together  with  the  coordinates  of  the  Cypher’s  GPS  antenna  and  the  radar  altimeter 
readings  to  calculate  the  XYZ  coordinates  of  each  sensor  at  the  instant  of  each 
measurement.  The  corrected  data  were  then  gridded  and  displayed  as  color-coded 
contour  maps  using  the  Surfer  (Golden  Software,  Inc.)  contouring/mapping  program. 
Numerical  estimates  of  target  locations  and  sizes  were  then  obtained  by  applying  the 
Geometrics/AETC  magnetic  anomaly  identification  and  detection  program,  MagAID. 

2.1.7  SURVEY  PROCEDURES 

The  areas  (scenarios)  to  be  covered  in  this  demonstration  survey  were: 

•  Artillery  and  Mortar  Range,  3.9  hectares  (Scenario  2) 

•  Interrogation  and  Burial  Sites,  3.9  hectares  (Scenario  4) 

•  Aerial  Gunnery  Range,  3.9  hectares  (Scenario  1) 

The  approach  was  to  fly  preprogrammed  search  patterns  along  sets  of  roughly  parallel 
lines  spaced  approximately  3  m  apart.  Data  were  collected  at  the  rate  of  20  samples/s  per 
channel.  At  a  flight  speed  of  3  m/s,  this  yielded  approximate  data  densities  of  6.6  data/m 
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along  the  flight  lines  and  .7  datum/m  in  the  direction  perpendicular  to  the  flight  lines. 
The  aircraft  was  flown  as  low  as  possible,  usually  at  a  height  of  less  than  3  m,  to 
maximize  target  detectability.  For  a  variety  of  reasons,  it  was  not  always  possible  to 
utilize  automated  flying.  In  those  cases,  the  aircraft  was  manually  controlled. 

A  proton  precession  magnetometer  was  used  to  recorded  the  diurnal  variations  in  the 
ambient  magnetic  field.  This  sensor  was  placed  at  the  same  fixed  location  during  each 
survey  day  and  recorded  data  at  30-sec  intervals.  The  variations  were  subtracted  from  the 
Cypher-measured  data. 


2.2 

RESULTS 

2.2.1 

Summary 

During  the  5-day  demonstration  period,  a  total  of  3.0  hectares  in  Scenarios  2  and  4  were 
surveyed  in  4.6  hr  of  flight  time.  When  the  aircraft  was  functioning  properly,  it 
efficiently  surveyed  large  open  areas  at  a  rate  of  approximately  .5  hectare  per  20-minute 
flight. 

Thirty-one  targets  are  reported  as  detections  (Figure  2).  These  are  distributed  more  or 
less  uniformly  over  the  surveyed  areas  and  are  assumed  to  be  targets  that  are  relatively 
large  and/or  buried  at  shallow  depths. 

2.2.2  Problems  Experienced 

1)  Mechanical  and  electrical.  During  the  test  period,  the  aircraft  experienced  several 
unusual  problems  that  limited  the  flight  time  available  for  data  acquisition. 
Unfortunately,  troubleshooting  of  the  aircraft  took  up  the  majority  of  the  time 
available  for  the  survey.  The  aircraft  experienced  engine  electronics  problems  as  well 
as  problems  with  high  drifts  in  the  velocities  provided  by  the  inertial  navigation 
system.  Strong  winds  from  unfavorable  directions  and  the  number  of  trees  on  the 
demonstration  site  resulted  in  inefficient  search  patterns. 

2)  Altitude  control.  The  radar  altimeter  currently  used  on  the  Cypher  is  not  accurate  at 
altitudes  of  less  than  3  m,  whereas  high-sensitivity  UXO  detection  requires  sensor 
heights  of  less  than  2  m.  The  known  limitation  of  the  altimeter  and  a  lack  of  terrain¬ 
following  software  made  the  Sikorsky  personnel  reluctant  to  fly  at  altitudes  low 
enough  to  achieve  good  target  detectability. 

3)  Automated  flight  manager.  The  software  that  was  available  for  automated  survey 
patterns  was  limited  to  rectangular  areas.  This  resulted  in  inefficient  operation  and 
incomplete  coverage  where  the  presence  of  trees  required  odd-shaped  areas.  There 
was  also  a  lack  of  confidence  in  the  ability  of  the  computer  and  the  flight  control 
system  to  fly  close  to  the  trees  even  though  we  used  a  mobile  NovAtel  GPS  unit  to 
establish  accurate  tree  boundaries. 
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4)  Magnetic  noise.  It  was  not  possible  to  correct  for  all  of  the  magnetic  noise  associated 
with  changes  in  the  attitude  of  the  aircraft.  Rapid  changes  in  heading  seemed  to 
produce  transients  that  were  difficult  to  correct.  The  residual  average  noise  level  was 
only  a  few  nT,  but  noise  spikes  of  20  nT  or  more  produced  fragmented  anomalies  and 
resulted  in  spurious  anomalies  that  had  amplitudes  comparable  to  those  expected  to 
be  associated  with  real  targets. 

2.2.3  Discussion 

The  performance  of  the  system  in  terms  of  both  operations  and  target  detection  was 
substantially  less  than  anticipated  and  was  disappointing.  On  the  other  hand,  the 
deficiencies  in  the  still-evolving  system  were  clearly  illuminated  and  were  found  to  be  of 
a  type  that  can  be  remedied  by  engineering.  No  assaults  on  the  fundamental  principles  of 
physics  appear  to  be  needed  to  achieve  improved  performance.  The  needed 
improvements  follow  directly  from  the  problems  enumerated  above.  Continued 
development  of  the  aircraft  will  enhance  its  reliability  and  its  controllability  in  windy 
conditions.  A  more  significant  issue  is  the  ability  to  fly  lower,  or  at  least  to  ensure  that 
the  sensors  can  be  flown  closer  to  the  ground  surface.  This  will  require  an  improved  low- 
altitude  altimeter  and  terrain-following  software  and  will  be  closely  tied  to  an  overall 
improvement  in  the  automated  flight  manager.  A  further  reduction  of  magnetic  noise  can 
be  achieved  by  reducing  the  mass  of  ferrous  components  in  the  aircraft  and  by  improved 
measurement  and  handling  of  aircraft  attitude  effects.  The  necessary  effort  seems 
worthwhile  in  view  of  the  safety  and  efficiency  benefits  to  be  gained  by  using  this  type  of 
sensor  platform  in  large-scale  UXO  surveys. 
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Banelle  -  Combined  Statistics:  Scenario  2  oniy 


Detection  Statistics  (TMA*  Group,  Sm  criticai  radius) 


#  Baseline 

Pd® 

P  random 

Ordnance 

67 

8 

0.12 

0026 

Nonordnance 

50 

5 

0.10 

Total 

117 

13 

Number  False  Alarms 

8 

False  Alarm  Rate  (#/Hectare) 

1.71 

False  Alarm  Ratio  (#/Ord.) 

1.00 

Probability  False  Alarms 

0.013 

Localization  Statistics  (TMA  Closest,  5m  critical  radius)  (in  meters) 


Mean 

Std  Deviation 

dx  •  eastinq  error 

-1.45 

2.24 

dv  -  northinq  error 

0.64 

2.39 

Radial  error 

3.37 

1.07 

Deoth  (z)  1 

dz  -  averaged  depth  error 

0.57 

0.83  1 

Idzl'-  absolute  depth  error 

0.98 

Characterization  Statistics  (TMA  Closest,  5m  critical  radius^ 


#  Baseline 

#  Detected 

Pd“ 

#  Correct 

Abilitv  to  Type 

Ordnance 

67 

8 

0.12 

0 

0.00 

Nonordnance 

50 

5 

0.10 

0 

0.00 

Larqe 

3 

0 

0.00 

0 

Medium 

31 

4 

0.13 

0 

0.00 

Small 

33 

4 

0.12 

0 

0.00 

Abilitv  to  Classify  - 1 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

4 

0.10 

0 

0.00 

Mortar 

26 

4 

0.15 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
“Probability  of  correct  characterization 
NA  •  not  applicable 
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Depth  (meters) 


3.0  CHEMRAD’ 

Chemrad  demonstrated  from  October  9  through  13,  1996  at  the  16-hectare  area  at  JPG. 
Chemrad  also  participated  in  Phase  I  of  the  UXO  ATD  program. 

3.1  TECHNOLOGY  DESCRIPTION 

3.1.1  Sensor  System  and  Transport  Mode 

The  Survey  system  used  for  the  JPG  HI  tests  comprised  of  3  major  Subsystems 

1.  The  Geonics  EM61 

2.  The  Geonics  EM61-3D  (3-component  time  domain  EM  system) 

3.  the  USRADS  2200+  for  ultrasonic  and/or  DGPS  positioning  as  desired 


The  Geonics  EM61  is  a  high  sensitivity  high  resolution  time-domain  metal  detector.  It 
consists  of  a  transmitter  that  generates  a  pulsed  primary  magnetic  field,  which  induces 
eddy  currents  in  nearby  metallic  objects.  The  decay  of  the  currents  is  measured  by  two 
receiver  coils  mounted  on  the  coil  assembly  and  output  as  two  separate  channels  of  data. 


^  CHEMRAD,  Final  Report  for  the  Advanced  Technology  Program,  Phase  HI  - 14 
November  1996 
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The  EM61-3D  also  consists  of  a  powerful  transmitter  that  generates  a  pulsed  primary 
magnetic  field  which  induces  eddy  current  in  nearby  metallic  objects.  The  time  delay  of 
these  currents  is  accurately  measured  over  a  wide  dynamic  range  of  time,  in  three 
orthogonal  space  components.  The  output  of  each  sensor  is  measured  and  recorded  by 
the  main  console  at  20  geometrically  spaced  gates,  covering  a  time  range  from  320 
microseconds  to  32  milliseconds.  The  EM61-3D  was  developed  to  provide  an  improved 
method  to  locate  and  characterize  UXO. 

The  USRADS  2200+  ties  the  systems  together  by  providing  location  information  and 
matting  this  spatial  information  with  the  continuously  recorded  data  from  the  sensors  and 
in  the  case  of  the  EM61,  transmits  the  data  via  RF  to  the  USRADS  2200+  Control  Center 
at  1  Hz.  The  USRADS  2200+  Control  Center  will  display  color  track  maps  showing  the 
real-time  position  and  sensor  output  to  aid  in  survey  navigation  and  support  immediate 
re-acquisition  of  suspect  findings.  The  USRADS  2200+  has  both  DGPS  and  ultrasonic 
positioning  capabilities  built  into  it’s  design  to  assure  all  terrain  survey  coverage.  Since 
the  JPG  site  had  very  sparse  tree  cover,  the  DGPS  capability  was  used  exclusively  for  this 
survey. 

The  USRADS  2200  has  been  repeatedly  man-carried  by  back-pack  using  a  three  person 
team  into  ex^emely  rugged  terrain  and  successfully  operated  for  up  to  8  hours  per  day. 
The  EM61-3D  and  supporting  electronics/battery  pack  can  be  carried  by  back-pack  with 
at  least  a  two  person  team.  The  EM61  and  EM61-3D  are  both  man  towable  surveying 
platforms. 

3.1.2  Recommended  Applications  and  Technology  Limitations 

The  primary  problem  facing  the  UXO  detection  arena  today  is  the  discrimination  of  UXO 
targets  from  non-UXO  targets.  Pulsed  EM  Induction  (PEMI)  techniques  have  been 
demonstrated  to  provide  significant  discrimination  ability  via  the  use  of  high  power  EM 
transmission  and  the  resultant  strong  eddy  currents  induced  in  the  targets.  The 
subsequent  reflected  EM  signal  from  a  metallic  target  is  generally  much  greater  than  that 
resulting  from  the  ground  and  thereby  easily  detected.  Even  weak  signals  can  be 
discriminated  since  the  ground  response  can  easily  be  modeled  as  a  stable  term.  More 
recently,  time  gating  of  the  reflected  signal’s  induced  current  in  the  receiving  sensor  has 
been  demonstrated  to  be  characteristic  of  the  size  and  shape  of  the  UXO  target  (McNeill 
and  Bosnar,  1996,  and  Kaczkowski  and  Gill,  1996). 

The  limitations  of  the  EM61  systems  are  that  they  are  have  a  maximum  depth  penetration 
of  approximately  3  meters  and  since  the  EM61-3D  is  a  prototype  unit,  actual  target  data 
for  evaluating  the  effectiveness  of  the  system  has  not  existed  before  this  survey.  Another 
associated  limitation  because  the  system  is  a  prototype  is  that  processing  software  is 
unavailable  or  very  limited  in  capability. 
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3.1.3  Logistical  System  Requirements 


The  USRADS  2200+  and  the  EM61  are  man-portable  and  are  both  designed  to  operate  at 
remote  sites  in  rugged  terrain  for  up  to  8  hours  per  day.  Hardware  components  require 
overnight  recharging  with  115V  AC.  The  system  Control  Center  is  routinely  operated 
from  a  4  wheel  drive  vehicle.  Field  manpower  requirements  include  2  surveyors  and  1 
operator  per  team. 

3.1.4  Data  Acquisition,  Processing,  and  Interpretation 

The  data  acquisition  was  performed  in  two  parts.  A  standard  EM61  interfaced  to  the 
USRADS  2200+  System  was  used  to  survey  the  grid  at  a  survey  rate  of  Im  per  second  on 
parallel  lanes  separated  by  1.5m.  This  corresponds  to  a  preliminary  survey  rate  of 
approximately  0.4  hectares  (1  acre)  per  hour  including  field  QA/QC.  These  anomalous 
target  findings  were  identified  with  the  real-time  data  presentation  system  and  suspect 
area’s  were  resurveyed  with  the  EM61-3D.  The  EM61  was  used  to  scan  the  survey  grid 
to  determine  the  targets  “X-Y”  location  and  depth  information.  The  EM61-3D  system 
was  then  used  to  attempt  determination  of  the  targets  shape  characteristics,  alignment  and 
classification.  The  EM61  data  was  processed  using  CHEMRADS  “ANALYZE®” 
Program.  Tfie  EM61-3D  data  was  downloaded  and  processed  using  a  modified  Protem 
processing  package  supplied  by  Geonics  Limited.  All  data  Acquisition,  and  processing 
were  performed  on  486  or  Pentium-based  computers  running  standard  Windows  3.1 
Software. 

The  Demonstrator  Reference  Area  (DRA)  was  processed  first  and  provided  a  high  degree 
of  confidence  for  the  EM61  data  interpretation.  The  data  collected  with  the  EM61-3D 
contained  multiple  noise  spikes  and  so  far,  a  high  degree  of  data  uncertainty  when 
attempting  to  use  the  transform  response  for  determining  ordnance  classification.  A 
review  of  the  current  data  is  still  in  process  as  is  the  writing  of  new  computer  codes  to 
more  efficiently  filter  and  process  the  3D  data. 


3.2  DEMONSTRATION  RESULTS 
3.2.1  Assumptions  and  Problems  Encountered 

1.  Assumption:  CHEMRADS  survey  would  occur  from  October  23  to  October  29, 
1996.  Actual:  Unfortunately,  CHEMRAD  discovered  that  Geonics  Limited  had 

concurrently  rented  the  one  EM61-3D  prototype  system  to  another  company  that  was 
scheduled  on  the  JPG  grid  during  the  exact  same  time  period  as  CHEMRADs’  Survey 
slot.  To  alleviate  this  situation,  CHEMRAD  accelerated  it’s  project  plan  to  perform  the 
survey  in  the  only  other  time  slot  available,  which  advanced  the  timeline  by  two  weeks. 
This  schedule  acceleration  adversely  affected  the  following  items: 

As  soon  as  the  3D  system  was  built,  it  was  shipped  to  CHEMRAD  for  use  on  the  JPG 
Project.  There  was  very  little  testing  of  the  equipment  electronics  and  equipment 
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operation  prior  to  it’s  first  actual  field  use  at  JPG.  A  one  week  trip  by  CHEMRAD  to 
Geonics  and  Geosoft  to  test,  and  modify  the  system  for  proper  field  operation  and 
software  program  performance  had  to  be  canceled  since  it  conflicted  with  the  new  survey 
dates.  Collection  and  processing  of  3D  data  for  signature  analysis  from  other  test  sites  in 
the  Eastern  US  were  canceled  due  to  schedule  acceleration. 

As  with  most  prototypes,  several  complications  were  discovered  just  prior  to  receipt  of 
the  3D  system,  but  with  the  shortening  of  the  available  schedule,  there  was  no  time  to 
improve  the  system  prior  to  the  JPG  survey.  The  following  are  examples  of  ways  in 
which  the  3D  system  can  be  or  was  improved  following  our  JPG  project. 

The  3D  system  did  not  have  an  active  data  output  port  which  would  enable  telemetry  of 
the  data  to  a  Personal  Computer  (PC)  for  real  time  data  processing  and  analysis.  This 
would  remove  the  painfully  slow  data  downloads  currently  required  by  the  equipment  and 
improve  Quality  Control  (QC)  of  the  collected  data.  There  was  also  no  way  to  review  or 
look  at  the  data  without  downloading  to  a  computer.  This  prohibited  the  surveyor  from 
efficiently  modifying  gains,  stacking  data,  or  determining  if  the  equipment  was  operating 
properly. 

After  collecting  data  with  the  3D  system  for  1.5  hours,  data  had  to  be  downloaded  (which 
would  take  1  hour)  before  proceeding  with  further  data  collection.  Geonics  has  reduced 
the  download  time  considerably  since  CHEMRADs  JPG  survey,  unfortunately  the 
software  and  EPROMS  to  upgrade  the  system  arrived  the  day  after  CHEMRAD 
completed  surveying  the  JPG  grid.  Ability  to  externally  trigger  the  3D  system  and  to 
easily  synchronize  the  location  system  with  the  3D  data.  A  time  based  system  was 
instituted  to  resolve  this  problem. 

The  weight  and  bulk  of  the  3D  system  needs  reduced  so  that  it  weighs  a  lot  less  than  it’s 
current  60+  lbs  to  reduce  worker  fatigue.  The  3D  system  has  an  intermittent  problem  that 
corrupted  large  amounts  of  data  but  would  then  appear  to  collect  good  data  for  a  few 
targets  and  then  return  to  collecting  sections  of  unusable  data.  Originally,  it  was  believed 
that  we  were  seeing  interference  from  another  demonstrator  nearby  who  was  using  a  high 
powered  EM  transmitter  system  but  there  are  area’s  of  the  grid  where  the  second 
demonstrator  would  have  been  at  least  400  feet  away  from  the  3D  system  and  the  data 
collected  is  unusable.  Potential  sources  for  the  problem  range  from  an  intermittent 
electronic  failure  to  interference  from  nearby  high-powered  transmitters  or  jammers  used 
by  the  military  during  the  practice  bombing  runs  to  coupling  with  the  Receiver  coils  or 
interference  with  the  electronics  carried  in  the  backpack. 

3.2.2  Technology  Conclusions 

The  USRADS+  DGPS  System  and  the  standard  EM61  were  very  dependable  and 
provided  an  excellent  method  of  quickly  scanning  the  combined  20  acres  of  Scenario’s  1 
and  2  to  determine  where  area’s  of  interest  exist.  It  was  at  this  point  that  the  EM61-3D 
system  was  to  be  used  to  discriminate  between  buried  ordnance  and  non-ordnance  items. 
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Theoretical  and  small  scale  tests  (McNeill,  J.D.  and  M.  Bosnar  1996)  have  shown  that  the 
EM61-3D  system  can  delineate  between  the  following  ordnance  items: 

40  mm  shell 
60  mm  M2  mortar 
81  mm  mortar 
105  nun  M 14  shell 
155mmM107  shell 

Unfortunately,  at  the  time  of  the  Phase  IH  Advanced  Technology  Demonstration,  the 
system  appears  not  to  be  either  durable  enough  for  field  use  or  have  proper  shielding  to 
provide  consistent  performance.  Additional  improvements  required  are: 

•  providing  an  active  data  output  port  similar  to  the  EM61 
•Reducing  the  bulk  of  the  instrument  package 

•Improving  basic  software  available  for  processing  so  that  the  data  profiles 
larger  than  3  minutes  worth  of  data  collection  can  be  handled  more  efficiently. 

Once  these  ijnprovements  are  made,  the  EM61-3D  system  should  provide  the  needed 
discrimination  capability  to  improve  the  accuracy  of  UXO  Surveys. 
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CHEMRAD  -  Combined  Statistics:  Scenarios  1,2 


Detection  Statistics  (TMA*  Group) 


#  Detected 

Pd^ 

Ordnance 

110 

55 

0.50 

0.024 

Nonordnance 

127 

50 

0.39 

Total 

237 

105 

Number  False  Alarms 

105 

False  Alarm  Rate  (#/Hectare) 

12.90 

False  Alamn  Ratio  (#/Ord.) 

1.91 

Probability  False  Alarms 

0.016 

Locail2atlon  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation 

Position  (x,v) 

dx  -  eastinq  error 

0.33 

0.83 

dy  -  northing  error 

0.02 

0.79 

Radial  error 

1.08 

0.50 

Depth  (z) _  .. 

dz  -  averaged  depth  error 

0.52 

0.43 

IdzP-  absolute  depth  error 

0.68  || 

Characterization  Statistics  (TMA  Closest) 


1  #  Baseline  |  #  Detected  I  Pd“  |#  Correct)  Pc° 

Abiiitv  to  Type 

Ordnance 

110 

55 

0.50 

0 

0.00 

Nonordnance® 

128 

50 

0.39 

0 

0.00 

Large 

14 

6 

0.43 

0 

0.00 

Medium 

38 

18 

0.47 

0 

0.00 

Small 

58 

31 

0.53 

0  1 

0.00 

lAbilitv  to  Classify  1 

Bomb 

21 

9 

0.43 

0 

0.00 

Projectile 

41 

20 

0.49 

0 

0.00 

Mortar 

26 

9 

0.35 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

17 

0.77 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

Probability  of  detection 
^  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 
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Depth  (meters) 


4.0  ENSCO*^ 

Ensco  demonstrated  from  October  15  through  17,  1996  at  the  16-hectare  area  at  JPG. 

Ensco  also  participated  in  Phase  I  of  the  UXO  ATD  program. 

4.1  TECHNOLOGY  DESCRIPTION 

ENSCO  demonstrated  three  technologies:  i)  the  MagnaLog  magnetic  sensing  system,  ii) 

a  Sensors  &  Software  pulseEKKO  1000  GPR  system,  and  iii)  a  White  Spectrum  XLT 

metal  detector.  Each  will  be  discussed  in  turn  in  the  following  sections. 

4.1.1  Sensor  System  and  Transport  Mode 

MamaLos.  MagnaLog  is  a  hand-held,  digital  vertical  magnetic  gradient  sensor  array  and 
data  acquisition  system  developed  by  ENSCO.  Employing  two  Schonstedt  GA-72- 
CD  sensors  and  an  on-board  microcontroller,  Magn^og  allows  a  single  operator  to 
survey  an  acre  in  35-45  minutes  at  a  3-ft  sensor  separation.  The  data  collection  rate 
along  profile  lines  is  user  selectable,  but  we  t5q)ically  use  between  14  and  Vi  ft/sample 
(5-10  saq^ples/sec).  The  operator  wears  the  system  comfortably  from  the  shoulders. 
The  microcontroller  has  a  12-button  keypad  to  setup  data  collection  parameters. 
During  data  collection,  the  operator  only  has  to  use  a  single  push-button  to  interact 
with  the  system  by  marking  start  and  end  points  of  lines,  to  pause,  or  to  mark  an  item 
of  interest.  Position  data  is  acquired  by  walking  profile  lines,  typically  200-ft  in 
length,  with  a  marker  at  the  mid-point  of  the  line.  The  MagnaLoc  processing 
software  interpolates  data  positions  based  on  a  constant  walking  speed,  start  and  end 
positions  of  the  profile,  and  gaps  in  the  data  due  to  obstructions  (trees,  etc.). 


^  ENSCO,  Inc.,  Demonstration  of  Capability  to  Detect  Unexpioded  Ordnance 
at  the  Jefferson  Proving  Grounds  Phase  IH,  Advanced  Technology  Demonstration, 
November  21, 1996 
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pulseEKKO  1000.  The  pulseEKKO  1000  is  a  portable,  digital  ground-penetrating  radar 
system  manu^ctured  and  marketed  by  Sensors  &  Software,  Inc.  The  pulseEKKO  1000 
can  be  operated  by  a  single  user  who  carries  the  electronics  in  a  specially  configured 
backpack.  For  this  effort,  we  transported  the  system  in  a  garden  wheel-barrow  to 
locations  where  we  would  collect  data.  Then,  data  are  acquired  by  dragging  the  antenna 
pair  along  the  ground. 

Spectrum  XLT.  The  Spectrum  XLT  is  an  off-the-shelf,  hand-held  metal  detector.  It  is 
operated  by  sweeping  the  ground  over  suspected  targets  and  listening  for  a  “squawk” 
that  indicates  a  conductive  target.  The  operator  notes  the  position  of  the  sensor  when 
he  hears  the  “squawk”. 

4.1.2  Recommended  Applications  and  Technology  Limitations 

MamaLos.  MagnaLog  is  designed  for  rapid,  inexpensive  magnetic  gradient  surveying.  It 
is  particulary  effective  for  detection  of  shallow  (<  5  m)  ferrous  objects.  As  a 
magnetometer  system,  MagnaLog  will  not  detect  non-ferrous  objects.  MagnaLog  can 
operate  in  most  any  climatic  conditions. 

pulseEKKO  1000.  As  a  GPR  system,  the  pulseEKKO  1000  is  used  for  characterizing  of 
subsurface  conditions.  GPR  is  most  effective  in  resistive  soils.  Conductive  soils 
contribute  to  signal  attenuation.  GPR  can  be  unusable  in  highly  conductive  soils.  For 
UXO,  GPR  is  most  effective  for  validating  and/or  characterizing  previously  detected 
anomalies.  The  pulseEKKO  1000  has  not  been  ruggedized  for  use  in  rain  or  freezing 
temperatures. 

Spectrum  XLT.  Designed  for  use  primarily  as  a  “coin  finder”  or  “treasure  hunter”,  the 
Spectrum  XLT  is  widely  used  in  the  EOD  industry  for  detecting  shallow  metallic 


objects.  The  depth  of  investigation  of  the  Spectrum  XLT  is  approximately  1-ft.  It 
detects  both  ferrous  and  nonferrous  objects.  It  operates  only  in  analog  audio  mode; 
no  digital  data  is  acquired. 

4.1.3  Logistics  Requirements 

MamaLos.  MagnaLog  is  hand-carried  by  a  single  operator  and  runs  on  battery  power. 

pulseEKKO  1000.  The  pulseEKKO  1000  can  be  operated  by  a  single  user  in  a  backpack 
mode.  We  chose  to  place  the  system  in  a  portable  wheel  barrow.  The  system  is 
battery  powered.  It  can  be  operated  by  either  one  or  two  people. 

Spectrum  XLT.  The  Spectrum  XLT  is  a  single  user,  hand-held  system.  It  is  battery 
powered. 

4.1.4  Data  Acquisition 

MasnaLos.  MagnaLog  acquires  digital  data  continuously  while  the  operator  walks  a 
profile  line.  Data  are  downloaded  periodically  to  a  portable  computer. 

pulseEKKO  *  X 000.  The  pulseEKKO  1000  allows  data  collection  in  profile  lines.  We 
used  the  system  to  acquire  data  only  over  specific  targets. 

Spectrum  XLT.  The  Spectmm  XLT  provides  only  audio  data. 

4.1.5  Data  Processing  and  Interpretation 

MasnaLos.  MagnaLog  data  is  processed  using  the  MagnaLog  data  processing  package 
and  the  LocPlot  plotting  routine.  We  prepare  2-D  contour  plots  of  magnetic 
anomalies.  We  also  display  each  profile  line  of  collected  data  (amplitude  versus 
position).  The  data  are  then  jointly  interpreted  from  these  two  displays,  though  the 
profile  plots  are  the  primary  interpretation  tool.  When  azimuths  are  estimated,  they 
are  based  on  the  spatial  orientation  of  the  anomaly  field. 

pulseEKKO  1000.  The  pulseEKKO  1000  data  are  plotted  as  profile  lines.  Display  signal 
processing  includes  gain  application,  removal  of  ensemble  averages,  etc. 
Interpretation  is  subjective  depending  on  the  displayed  data. 

Spectrum  XLT.  No  data  is  processed.  The  operator  interprets  results  in  real-time  based 
on  the  audio  data. 

4.1.6  Quality  Assurance 

MasnaLos.  MagnaLog  operating  condition  is  verified  at  the  beginning  and  end  of  each 
data.  In  addition  we  download  and  validate  data  approximately  every  hour  in  the 
field. 
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pulseEKKO  1000.  Quality  assurance  of  GPR  data  is  maintained  by  subjective 
interpretation  of  collected  data,  in  much  the  same  manner  as  the  target  data  is 
interpreted  subjectively. 

Spectrum  XLT.  The  Spectrum  XLT  is  validated  by  operating  it  over  surface  metallic 
objects. 

4.2  DEMONSTRATION  RESULTS 

ENSCO  conducted  a  demonstration  in  three  areas:  1)  Scenario  1.  Aerial  Gunnery  Range, 

2)  Scenario  2.  Artillery  and  Mortar  Range,  and  3)  Scenario  4.  Interrogation  and  Burial 

Area.  Each  of  these  three  scenarios  will  be  discussed  in  turn. 

4.2.1  Assumptions 

Scenario  1 .  Aerial  Gunnery  Range.  We  made  no  assumptions  other  than  those  provided 
in  the  Demonstration  Work  Plan. 

Scenario  2.  Artillery  and  Mortar  Range.  We  made  no  assumptions  other  than  those 
provided  in  the  Demonstration  Work  Plan. 

Scenario  4.  Interrogation  and  Burial  Area.  We  made  no  assumptions  other  than  those 
provided  in  the  Demonstration  Work  Plan. 

4.2.2  Site-Specific  Procedures 

Scenario  1.  Aerial  Gunnery  Range.  We  collected  MagnaLog  data  in  approximately  1- 
acre  segments  (61-m  by  61-m.)  Plastic  flagging  was  used  to  mark  walking  profiles. 
Data  were  collected  in  profiles  spaced  by  6-ft  (1.8  m),  which,  with  our  3-ft  (0.9  m) 
sensor  separation,  allowed  data  to  be  acquired  on  3-ft  (0.9  m)  profile  spacing.  Data 
were  acquired  to  overlap  adjacent  grids  by  at  a  minimum  5  feet.  Data  were  collected 
at  a  rate  of  5  samples/sec,  which  corresponds  to  an  average  spatial  sampling  rate  of 
0.5  ft  (15  cm).  Data  were  acquired  moving  in  a  north-south  direction.  Following  data 
analysis,  as  many  anomalies  as  time  permitted  were  reoccupied  so  that  their  position 
and  character  could  be  validated. 

Scenario  2.  Artillery  and  Mortar  Range.  We  collected  MagnaLog  data  in  approximately 
1-acre  segments  (61-m  by  61-m.)  Plastic  flagging  was  used  to  mark  walking  profiles. 
Data  were  collected  in  profiles  spaced  by  6-ft  (1.8  m),  which,  with  our  3-ft  (0.9  m) 
sensor  separation,  allowed  data  to  be  acquired  on  3-ft  (0.9  m)  profile  spacing.  Data 
were  acquired  to  overlap  adjacent  grids  by  at  a  minimum  5  feet.  Data  were  collected 
at  a  rate  of  5  samples/sec,  which  corresponds  to  an  average  spatial  sampling  rate  of 
0.5  ft  (15  cm).  Following  data  analysis,  as  many  anomalies  as  time  permitted  were 
reoccupied  so  that  their  position  and  character  could  be  validated. 


B.4-4 


Scenario  4.  Interrogation  and  Burial  Area.  For  each  of  the  20  targets  we  were  assigned, 
we  setup  a  36-ft  by  36-ft  grid  centered  on  the  marked  position.  MagnaLog  data  were 
collected  on  1.5-ft  (0.46-m)  profiles  in  a  north-south  direction.  The  Spectrum  XLT 
was  used  to  investigate  whether  the  target  was  shallow,  and  if  so,  its  length  and 
orientation.  GPR  data  were  collected  along  four  profiles.  Each  profile  started  and 
stopped  at  the  edge  of  our  36-ft  grid.  The  profiles  were  collected  in  a  north-to-south, 
northwest-to-southeast,  northeast-to-southwest,  and  west-to-east  orientation.  Each  of 
these  four  profiles  crossed  directly  over  the  PRC-provided  marked  position. 

4.2.3  Problems  Encountered 

No  significant  problems  were  encountered.  Time  on  the  grid  as  determined  by  the  PRC 
EMI  representatives  was  closely  controlled  and  rigidly  adhered  to.  This  caused  some 
inefficiencies  in  data  acquisition  and  prevented  some  final  quality  assurance  data  from 
being  collected. 

Scenario  1.  Aerial  Gunnery  Range.  Data  collection  and  processing  required  longer  than 
expected^  Therefore,  we  were  only  able  to  reoccupy  a  limited  number  of  anomalies 
for  validation  and  relocation.  Because  of  the  high  quality  of  the  magnetic  data,  we 
focused  our  efforts  in  the  validation  stage  on  improving  location  estimates,  not 
improving  characterization.  We  relocated  91  anomalies  in  the  validation  stage.  Some 
of  the  (apparent)  larger  excavations  have  undergone  significant  settling,  which  causes 
uncertainty  in  determining  where  depth  is  measured  from.  Apparent  renmants  of  a 
fence  were  detected  in  this  scenario.  We  did  not  report  every  piece  of  the  fence  as  a 
target,  but  instead  reported  the  end  points  of  the  fence  as  a  target  (within  the  scenario 
boundaries),  and  so  annotated  the  comment  in  the  data  file. 

Scenario  2.  Artillery  and  Mortar  Ranee.  Data  collection  and  processing  required  longer 
than  expected.  Therefore,  we  were  only  able  to  reoccupy  a  limited  number  of 
anomalies  for  validation  and  relocation.  Because  of  the  high  quality  of  the  magnetic 
data,  we  focused  our  efforts  in  the  validation  stage  on  improving  location  estimates, 
not  improving  characterization.  We  relocated  79  anomalies  in  the  validation  stage. 
Apparent  remnants  of  a  fence  were  detected  in  this  scenario.  We  did  not  report  every 
piece  of  the  fence  as  a  target,  but  instead  reported  the  end  points  of  the  fence  as  a 
target  (within  the  scenario  boundaries),  and  so  annotated  the  comment  in  the  data  file. 
We  also  detected  a  large,  sheet-like  (mat-like)  target  in  this  scenario.  The 
approximate  mid-point  of  this  object  was  declared  a  target  and  so  annotated  in  the 
data  file. 

Scenario  4.  Interrogation  and  Burial  Area.  No  problems  were  encountered  during  data 
collection.  Definition  of  the  scenario  in  the  Demonstration  Work  Plan  was  not 
sufficiently  complete.  The  range  of  targets  for  this  scenario  were  not  adequately 
defined.  The  data  entry  format  (the  Excel  worksheet)  had  an  inadequate  range  of 
acceptable  entries.  For  example,  there  was  no  means  by  which  to  identify  a  bum  or 
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burial  site.  Also,  some  of  the  pin  flags  marking  locations  did  not  seem  to  be  located 
with  sufficient  accuracy,  which  was  a  “given”  for  the  scenario.  (This  might  have  been 
due  to  the  metal  pin  flags  used  to  mark  locations:  each  demonstrator  had  to  move  the 
ferrous  flags  to  survey  the  site.) 

4.2.4  Discussion  of  Results 

Scenario  1.  Aerial  Gunnery  Ranee.  We  believe  the  data  quality  was  high  and  the  results 
very  good.  The  MagnaLog  system  appeared  to  perform  well.  We  would  have  like  to 
have  had  the  time  to  validate  more  anomalies  with  GPR,  but  available  time  precluded 
this  effort. 

Scenario  2.  Artillery  and  Mortar  Range.  We  believe  the  data  quality  was  high  and  the 
results  very  good.  The  MagnaLog  system  appeared  to  perform  well.  We  would  have 
like  to  have  had  the  time  to  validate  more  anomalies  with  GPR,  but  available  time 
precluded  this  effort. 

Scenario  4.  Interrogation  and  Burial  Area.  Detection  and  location  were  successful.  GPR 
data  was  generally  of  high  quality:  targets  were  clearly  seen.  Only  one  target  was 
clearly  seen  with  the  Spectrum  XLT,  the  other  targets  being  too  deep. 


4.2.5  Raw  Data 

Digital  raw  data  are  attached  on  3.5  inch  IBM-format  diskettes.  Magnetic  data  are  found 
in  the  .csv  files;  GPR  data  are  located  in  the  .dtl  and  .hd  files.  All  data  files  have  been 
compressed  using  the  PKZIP  utility.  The  magnetic  data  files  contain  both  raw  and 
processed  data.  Raw  data  is  found  in  column  3. 

4.2.6  Processed  Data 

Processed  magnetic  data  are  included  in  the  magnetic  data  files  along  with  the  raw  data. 
Processing  includes  attachment  of  X  and  Y  coordinates,  computation  of  slowly  varying 
biases  and  trends  in  the  data,  and  extraction  of  a  residual  signal.  GPR  data  are  only 
processed  for  display  and  hence  are  not  included  here. 

4.2.7  Conclusions 

We  make  the  following  conclusions  from  our  demonstration: 

MagnaLog  is  an  effective  tool  for  detecting  and  locating  UXO.  It  is  efficient, 
inexpensive,  and  accurate. 
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MagnaLog  position  data  could  be  improved.  Our  analysis  indicates  the  majority  of  our 
position  error  is  in  the  direction  we  walked  the  instrument  (N-S),  with  minimal  error 
in  the  E-W  direction. 

GPR  can  be  an  effective  tool  for  validating  previously  detected  anomalies  and 
contributing  to  the  characterization  of  anomalies.  Even  for  the  well-known  poor  quality 
soils  at  JPG,  our  GPR  data  was  quite  good.  Many  previous  demonstrators  at  JPG  have 
shown  that  GPR  is  ineffective  at  UXO  detection  at  JPG.  We  believe  the  proper  role  for 
GPR  is  not  for  detection,  but  instead  to  validate  previously  detected  anomalies  and 
contribute  to  the  classification  of  the  target. 
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ENSCO  •  Combined  Statistics:  Scenarios  1,2 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

pd‘ 

P  random 

Ordnance 

110 

77 

0.70 

0.071  1 

Nonordnance 

127 

82 

0.65 

Total 

237 

159 

Number  False  Alarms 

396 

False  Alarm  Rate  (#/Hectare) 

48.66 

False  Alarm  Ratio  (#/Ord.) 

5.14 

Probability  False  Alarms 

0.061 

Localization  Statistics  (TMA  Closest)  (In  meters) 


Mean 

Std  Deviation  \\ 

dx  -  eastina  error 

-0.12 

0.32 

dv  -  northing  error 

0.05 

0.56 

Radial  error 

0.50 

0.43 

Deoth  fz) 

dz  -  averaged  depth  error 

-0.06 

0.29 

ldzl‘-  absolute  depth  error 

0.30 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Po*’ 

#  Correct 

Pc- 

F  v^iii7iri^»rT^iii^i^^^i^— 1— — 

Ordnance 

110 

77 

0.70 

0 

0.00 

Nonordnance® 

128 

82 

0.64 

0 

0.00 

Large 

14 

14 

1.00 

0 

0.00 

Medium 

38 

26 

0.68 

0 

0.00 

Small 

58 

37 

0.64 

0 

0.00 

Abilitv  to  Classify 

Bomb 

21 

21 

1.00 

0 

0.00 

Projectile 

41 

33 

0.80 

0 

0.00 

Mortar 

26 

14 

0.54 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

9 

0.41 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
‘Square  root  of  the  mean  square  depth  error 
Probability  of  correct  characterization 
*TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  •  not  applicable 
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■  Target  Detected 
□  Target  Not  Detected 


5.0  FOERSTER" 

Foerster  demonstrated  from  October  2  through  5,  1996  at  the  16-hectare  area  at  JPG. 
Foerster  also  participated  in  Phase  I  of  the  UXO  ATD  program. 

5.1  TECHNOLOGY  DESCRIPTION 


5.1.1  Sensor  System  and  Transport  Mode 

The  FORSTER  MULTI-CAT  is  a  towed  vehicle  array,  utilizing  the  best  in  fluxgate 
magnetometer  technology.  The  system  uses  up  to  9  new  patented  FORSTER  Sensors, 
developed  and  based  on  the  current  FEREX  sensor  which  is  in  service  in  the  US  Military 
and  known  as  the  Mark  26  Ordnance  Locator.  The  gradient  magnetometer  sensor 
principle  of  the  FEREX  is  widely  known.  It  is  most  suitable  for  the  detection  and  exact 
location  of  unexploded  bombs,  pipes,  cables,  ferromagnetic  containers,  etc.  Each  sensor 
assembly  contains  two  FORSTER  Probes  which  are  optimized  for  the  detection  of  ferrous 
objects  located  below  the  ground  surface.  Depending  on  the  size  and  position  of  the 
ferromagnetic  objects,  the  advanced  updated  FEREX-sensors  detect  ordnance  down  to  a 
depth  of  approximately  six  (6)  meters.  The  new  developed  patented  FEREX  sensors  are 
light  weight  and  aligned  for  life.  They  do  not  need  re-calibration  as  with  the  MK  26 
sensors.  Their  detection  sensitivity  is  improved  by  approximately  20  %.  The  MULTI¬ 
CAT  is  constructed  of  entirely  advanced  composites.  No  metal  is  used  in  its  manufacture. 
The  MULTI-CAT’S  unique  probe  array  assembly  coupled  with  FORSTER’s  MAGNETO 
software,  provides  users  with  an  advanced  and  accurate  ordnance  location  system. 


®  Foerster,  Demonstration  Summary  Report,  30  October  1996 


B.5-1 


Global  Positioning  System  (GPS) 


The  vehicle  mounted  GPS -system  consists  of  the  GPS  antenna,  the  stationary  and  mobile 
receiver  and  a  laptop  type  computer  with  an  additional  landscape  software  program.  The 
landscape  software  enables  the  presentation  of  a  map  of  the  whole  search  area  with 
specific  markings  such  as  comer  or  grid  points.  It  can  also  provide  for  a  grid  system 
adapted  to  the  intended  tracks  of  the  veWcle;  or  ,  for  example;  it  can  adapt  its  grid 
systems  to  the  areas  designated  for  the  demonstration  as  indicated  in  the  request  for 
proposal. 

The  GPS  and  landscape  software  system  enables  the  graphical  presentation  of  “excluded” 
areas  such  as  obstacles  which  then  are  specifically  marked,  and  will  be  searched  by  the 
conventional  handheld  datalogging  method  with  the  FEREX  (MK  26),  for  later 
integration  into  the  vehicle  mounted  search  system  results. 

Since  the  used  DGPS  system  is  a  so  called  differential  system,  there  is  (separate  from  the 
mobile  GPS  receiver  on  the  vehicle)  a  stationary  reference  receiver  station  which  is 
linked  by  UHF  transmission  with  the  mobile  station.  The  geographical  position  of  this 
reference  station  placed  nearby  to  the  starting  point  of  the  search  must  be  exactly 
measured  an(f  conventionally  described. 


5.1.2  Recommended  Applications  and  Technology  Limitations 


Regarding  the  difficulties  of  detecting  and  classifying  buried  unexploded  ordnance,  in 
general,  the  gradiometer  principle  cannot  differentiate  between  ferrous  ordnance  and  non¬ 
ordnance  items.  This  is  due  to  the  physical  fact  that  the  signatures  of  such  items  can  be 
more  or  less  the  same,  depending  on  composition,  type,  shape,  depth,  history,  and 
configuration.  Only  in  areas  where  the  ground  has  not  been  contaminated  by  non¬ 
ordnance  items,  the  presence  of  ordnance  can  be  determined  with  near  accuracy.  Our  data 
reports  indicate  that  we  are  unable  to  classify  targets  by  type  and  class.  This  is  a  major 
limitation  of  this  type  of  technology. 

5.1.3  Logistics  Requirements 


The  MULTI-CAT  system  can  be  transported  on  a  trailer  or  in  a  van  to  all  areas  of  interest. 
The  system,  itself,  is  self  maintained  with  sufficient  spare  electric  power  requirements. 
The  MULTI-CAT  uses  off  the  shelf  parts  for  its  sensors.  Due  to  Foerster’s  patented 
tension-band  probe  arrangement,  the  sensors  used  in  the  MULTI-CAT,  never  need 
adjusting.  No  special  maintenance  of  the  system  is  required.  The  MULTI-CAT  is  a  self 
contained  unit  with  no  internal  moving  parts,  except  the  wheels.  It  is  well  suited  for  a 
variety  of  operational  conditions,  such  as  remote  areas  and  in  rugged  terrain.  It  can  be 
used  both  on  land  and  underwater.  The  MULTI-CAT  can  also  be  configured  with  deep 
search  sensors,  enabling  the  MULTI-CAT  to  perform  even  deeper  search  detection 
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operations.  The  MULTI-CAT  is  easily  assembled  in  the  testing  area,  requiring  no  longer 
than  60  minutes  to  assemble.  The  MAGNETO  software,  uses  a  standard  486  based 
microcomputer,  that  is  easily  obtainable  and  configured. 

The  MULTI-CAT  can  be  towed  by  any  vehicle  that  is  suited  to  the  terrain  of  the  survey 
involved.  Depending  on  terrain  conditions,  a  scanning  rate  of  2  meters  per  second  is 
considered  well  within  the  capabilities  of  the  platform  in  non-forested  areas.  In  addition, 
the  vehicle  that  is  used  to  tow  the  MULTI-CAT  is  used  as  a  platform  for  the  MAGNETO 
Magnetometer  Survey  Software.  Also,  the  GPS  antenna,  transmitter,  and  receiver  package 
is  mounted  on  the  MULTI-CAT  and  the  towing  vehicle  to  serve  for  the  accuracy  needed 
in  determining  the  location  of  buried  munitions. 


5.1.4  Data  Acquisition 


During  the  data  collection  survey  phase,  the  measurement  data  of  each  individual  sensor 
was  digitized  with  extreme  resolution  and  transferred  via  a  multiplex  station  to  a  personal 
computer  (PC)  system  mounted  in  the  towing  vehicle.  By  the  use  of  the  MAGNETO 
software,  the  ^magnetic  anomalies  in  the  ground  are  presented  on  the  towing  vehicle  PC 
screen  in  real  time  and  automatically  stored  for  later  interactive  interpretation  and 
calculation  as  individual  objects  or  object  configurations  on  another  PC  station. 

5.1.5  Data  Processing  and  Interpretation 

The  MAGNETO  Software  can  be  run  on  any  486  Class  Personal  Computer  or  Laptop. 
The  recommended  system  requirements  are:  486DX266  or  higher  processor,  16  Mb 
RAM,  with  at  least  30-40  megabytes  of  free  hard  disk  space.  Interpretation  and 
calculation  of  columns  in  the  target  data  are  as  follows 

Target 

The  objects  in  the  lists  have  a  sequential  number.  These  two  numbers  are  combined  to  the 
target-demomination  for  the  final  evaluation  in  the  worksheets.  Except  for  Scenario  4,  in 
which  target  numbers  were  given. 

Northing/easting 

For  the  local  field  coordinate  system  in  the  MAGNETO  software  “Monument  3”  has  been 
used  as  the  reference  point  (zero  point).  They  have  been  transformed  into  UTM- 
coordinates  by  shifting  and  rotating  the  object  coordinates. 

Depth  &  Weight 
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Is  equivalent  to  the  MAGNETO  software  for  calculation  of  the  object  depth.  The  mass 
has  been  calculated  using  the  specific  density  of  iron  and  the  volume.  The  mass  serves  as 
a  decision  criteria  for  the  three  different  weight  classes. 

Size 

The  three  different  size  classes  have  been  determined  according  to  the  object  diameter. 

Azimuth 

This  value  represents  the  declination  in  the  MAGNETO  software.  The  range  had  to  be 
recalculated  from  “-180°  to  +180°”  into  “0°  to  360°”. 

Declination 

Is  the  same  as  MAGNETO  software  inclination  field  values. 

5.1.6  Quality  Assurance 

Quality  assurance  on-site  was  maintained  throughout  the  demonstration  by  performing  a 
daily  reference  calibration  comparison  on  the  reference  training  grid  provided  by  PRC 
and  a  daily  check  of  the  DGPS  coordinate  system  used  given  the  reference  points  of  the 
testing  grid.  Daily  backups  of  the  data  were  performed  to  ensure  data  integrity. 

5.2  DEMONSTRATION  RESULTS 

5.2.1  Assumptions 

Our  team  came  prepared  to  handle  any  difficulties  that  could  be  encountered  during  our 
demonstration  period.  Of  major  concern  was  the  coordinate  information  given  to  us  by 
PRC  personnel.  The  accuracy  of  our  software  is  almost  fully  dependent  on  the  accuracy 
of  the  coordinates  used  for  reference  and  positioning.  Many  inquiries  to  on-site  and  off¬ 
site  PRC  personnel  assisted  us  in  ensuring  that  the  coordinates  given  were  indeed 
accurate. 

5.2.2  Site-Specific  Procedures 

Each  day  of  data  collection  on  the  Jefferson  Proving  Ground  site  was  started  with  a 
systems  check  of  the  MULTI-CAT,  and  DGPS.  The  vehicle  system  consists  of  a 
combination  of  a  towing  vehicle  and  the  MULTI-CAT.  The  vehicle  is  equipped  with  a 
differential  mode  Global  Positioning  System  (DGPS)  consisting  of  an  antenna,  mobile, 
and  a  stationary  receiver. 

Depending  on  the  surface  conditions  of  the  search  area,  the  towing  vehicle  was  driven  up 
and  down  with  a  speed  of  approximately  one  to  two  (1-2)  meters  per  second  (m/sec) 
along  straight  and  parallel  lines.  In  some  cases  rectangular  search  fields/driving  path  was 
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used  as  a  result  of  this  search  procedure.  Such  fields,  however,  were  joined  together  by 
the  MAGNETO  software.  In  the  south  east  comer  of  Scenario  1,  Aerial  Gunnery  Range, 
the  terrain  prevented  the  MULTI-CAT  from  being  able  to  survey  due  to  the  proximity  of 
the  trees.  The  survey  of  this  area  was  performed  manually  with  a  standard  FEREX  4.021 
(MK  26)  modified  with  a  datalogger  for  use  with  our  single  channel  MAGNETO  version. 
The  data  collected  by  manual  survey  was  then  transferred  and  combined  for  evaluation 
with  the  MULTI-CAT’S  multi-channel  MAGNETO  system  software. 

5.2.3  Problems  Encountered 

The  only  equipment  related  problem  encountered  during  the  data  collection  phase  of  the 
demonstration  occurred  on  the  first  day.  A  failure  of  the  radio  data  modem  was 
discovered.  Within  approximately  10  minutes,  the  problem  was  corrected  and  no  further 
collection  problems  were  identified  for  the  duration  of  the  demonstration. 

5.2.4  Discussion  of  Results 

In  Scenario  1,  Aerial  Gunnery  Range,  a  total  of  167  targets  were  identified  by  the 
MAGNETO  software  in  approximately  2.2  hectares  of  surveyed  terrain.  In  Scenario  4, 
Interrogation^and  Burial  Sites,  all  twenty  targets  locations  were  given  to  us  by  PRC 
personnel.  In  four  occasions,  multiple  targets  were  identified  for  a  given  target  number. 
Again,  the  state  of  the  technology  used  prevents  us  from  accurately  determining  type  and 
class  of  the  objects  detected.  These  two  scenarios  were  completed  within  twenty  hours. 
On-site  PRC  personnel  allowed  us  to  continue  our  survey  in  Scenario  2,  Artillery  and 
Mortar  Range,  to  fill  our  allotted  twenty-four  on-grid  time  frame.  We  successfully 
surveyed  an  additional  1.85  hectares,  and  identified  172  targets  within  that  survey.  Please 
refer  to  the  attached  target  data. 

5.2.5  Conclusions 

Foerster  Instruments  Incorporated,  together  with  our  teaming  partners,  was  completely 
convinced  that  our  approach  afforded  us  the  ability  to  complete  the  stated  objective  listed 
in  the  statement  of  work.  The  approach  of  interfacing  GPS  with  the  search  signals, 
allowed  us  to  maneuver  without  ground  measurement  within  the  site  area,  except  the  true 
position  of  the  comer  points  and  other  reference  markings  which  were  provided.  In 
parallel,  areas  which  are  not  accessible  with  the  towed  vehicle  array  (dense  trees,  etc.) 
were  searched  with  hand-held  conventional  MK  26  MAGNETO  equipment.  Our  total 
system  provides  for  the  integration  of  all  data  obtained. 

Foerster  was  contracted  to  perform  a  nonintrusive  detection  demonstration  using  a 
combination  of  the  FOERSTER  MULTI-CAT  Towed  Vehicle  Array ,  and  magnetometer 
sensors  for  a  minimum  survey  of  a  2  hectare  Aerial  Gunnery  Range,  and  a  minimum  2 
hectare  Interrogation  and  Burial  Area  for  a  variety  of  buried  ordnance  in  24  working 
hours,  depending  upon  the  terrain  selected  for  demonstration.  The  FOERSTER  MULTI¬ 
CAT  exceeded  this  requirement.  We  were  able  to  successfully  complete  the  above  survey 
well  within  the  time  limits  set.  In  addition,  we  were  able  to  survey  an  additional  1.85 
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hectares  of  Scenario  2,  Artillery  and  Mortar  Range,  and  have  included  our  results  as  part 
of  the  data. 
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Foerster  -  Combined  Statistics:  Scenario  1  only 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Pd” 

Ordnance 

43 

26 

0.60 

0.054 

Nonordnance 

77 

49 

0.64 

Total 

120 

75 

Number  False  Alarms 

126 

False  Alarm  Rate  (#/Hectare) 

36.46 

False  Alarm  Ratio  (#/Ord.) 

4.85 

Probability  False  Alarms 

0.458 

Localization  Statistics  (TMA  Closest)  (In  meters) 


Mean 

Std  Deviation  | 

dx  -  eastina  error 

-0.05 

0.35 

dy  -  northing  error 

-0.04 

0.3 

Radial  error 

0.35 

0.31 

dz  -  averaged  depth  error 

0.15 

0.22  II 

Idzl®-  absolute  depth  error 

0.26  1 

Characterization  Statistics  (TMA  Closest) 


1  #  Baseline  |  #  Detected  |  Pd”  I#  Correct  |  Pc"  I 

Ordnance 

43 

26 

0.60 

0 

0.00 

Nonordnance® 

78 

49 

0.63 

0 

0.00 

lAbilitv  to  Size  1 

Large 

11 

10 

0.91 

10 

1.00 

Medium 

7 

4 

0.57 

3 

0.75 

Small 

25 

12 

0.48 

1 

0.08 

Ability  to  Classify  1 

Bomb 

21 

16 

0.76 

0 

0.00 

Projectile 

0 

0 

NA 

0 

NA 

Mortar 

0 

0 

NA 

0 

NA 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

10 

0.45 

0 

0.00 

Notes:  *  Target  Matching  Aigorithm 

Probabiiity  of  detection 
°  Square  root  of  the  mean  square  depth  error 
"  Probabiiity  of  correct  characterization 
^MA  affects  how  nonordnance  baseiine  targets  are  counted 
NA  -  not  appiicabie 
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Depth  (meters) 


JPG  Controlled  Site:  Phase  ill 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Crhlcai  Radius:  2  meters 
Demonstrator:  Foerster 


Principal  Diameter  (millimeters) 


■  Target  Detected 
□  Target  Not  Detected 


6.0  GEO-CENTERS,  INC.  ^ 


Geo-Centers  demonstrated  from  September  24  through  28,  1996  at  the  16-hectare  area  at 
JPG.  Geo-Centers  also  participated  in  Phases  I  and  II  of  the  UXO  ATD  program. 


6.1  DEMONSTRATED  TECHNOLOGIES 

6.1.1  Sensor  System  and  Transport  Mode: 

STOLS®  is  the  only  GPS -integrated  simultaneous  magnetometer,  gradiometer,  and  EM61 
survey  system  in  the  world.  It  effectively  performs  three  distinct  geophysical  surveys  for 
the  price  of  one,  and  does  so  with  the  highest  data  density  presently  offered  in  the 
industry.  Features  include: 


•Vehicular,  surface  towed,  concurrent,  multisensor  capability 

•Total  field/gradiometer  magnetometer  array  (8  Geometries  822A  cesium  vapor 
magnetometers  with  0.5  meter  spacing  updating  at  20  Hz,  arrayed  four  over  four  with  a 
15"  vertical  separation,  on  a  towed  platform) 

•Electromagnetic  pulsed  induction  sensor  array  (3  Geonics  EM61,  half-meter  coils,  upper 
and  lower,  updating  at  10  Hz,  on  front-mounted  platform) 

•Tow  vehicle  and  sensor  platforms  designed  for  low  magnetic  and  pulsed  induction  self¬ 
signatures 

•Electronics  optimized  for  low  magnetic  and  electromagnetic  noise 

•Trimble  differential  GPS  with  Real  Time  Kinematic  (RTK)  for  real-time  3  to  5  cm 

precision  data  processing  center 

•Same-day  production  of  images  of  magnetic  and  EM  data 


^  GEO-CENTERS,  New 
Vehicular  Multisensor 
Array  Technology  for  UXO 
Detection  (Phase  III)  at 
Jefferson  Proving  Ground, 
Madison,  Indiana  -  Final 
Survey  Report  -  October 
1996 
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6.1.2  Recommended  Applications  and  Technology 

With  its  vehicular-based  simultaneous  acquisition  of  GPS -integrated  total  field 
magnetometer,  gradiometer,  and  EM61  data,  STOLS®  is  absolutely  unique  in  its  ability  to 
rapidly  traverse  large  amounts  of  area  and  provide  very  high  density,  high  resolution 
geophysical  imagery  in  a  cost-effective  manner.  STOLS®  has  been  successfully  deployed 
on  over  50  commercial  surveys  spanning  UXO,  HTRW,  landfill,  underground  storage 
tanks,  archeological,  and  utility  mapping  applications.  Magnetometers  are  not  optimally 
fielded  in  areas  where  there  are  high  concentrations  of  magnetic  rock  or  other  local  high- 
frequency  magnetic  gradients;  however,  magnetometers  augmented  with  the 
electromagnetic  pulsed  induction  sensor  array  alleviate  this  situation.  STOLS®  has  proven 
performance  in  a  wide  variety  of  topography  and  conditions.  It  can  negotiate  30  degree 
inclines,  and  with  14  inch  vehicle  ground  clearance  it  has  been  deployed  in  wet 
conditions  including  foot  deep  swamp  water.  Open  terrains  are  easily  traversed  at  5  to  10 
mph.  Average  speeds  of  20  mph  have  been  sustained  over  smooth  topography.  The 
system  has  been  deployed  in  a  full  range  of  outdoor  temperature  conditions  from  below 
30  F  to  over  120  F.  Even  with  its  front-mounted  pulsed  EM  coils  and  towed 
magnetometer  platform,  the  system  is  quite  maneuverable.  The  differential  GPS  can  be 
freely  deployed  worldwide,  limited  only  by  surface  features  blocking  satellite  visibility. 

6.1.3  Logistics  Requirements 

STOLS®  is  a  self-contained  survey  system  with  no  external  requirements  beyond 
adequate  parking  for  the  tractor-trailer.  The  system  arrives  on-site  complete  with 
generators,  spares,  computers,  and  all  necessary  tools  for  field  maintenance,  repairs,  and 
data  processing. 

6.1.4  DATA  ACQUISITION 

STOLS®  technology  is  a  trilogy  of  total  field  magnetometer,  total  field  gradiometer,  and 
electromagnetic  (EM)  induction  sensors.  The  data  acquisition  equipment  for  the 
magnetometers  includes: 

•Mil-spec  486  computer  with  16  megabytes  RAM 
•Proprietary  Windows®-based  data  acquisition  software 

•Geometries  822A  cesium  vapor  total  field  magnetometers  deployed  in  total  field 
and  gradiometer  configuration 

•Proprietary  magnetometer  interface  acquiring  8  channels  at  20  Hz 
1  Hz  DGPS  input  for  precise  positioning  of  magnetometer  data  (GPS  antenna 
located  over  center  magnetometer) 

•Time-synchronized  diurnal  variation  data  concurrently  recorded  on  separate 
reference  magnetometer  station 
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Data  acquisition  equipment  for  the  electromagnetic  sensors  includes; 

•Rugged  486  computer  with  4  megabytes  RAM 
•Proprietary  data  acquisition  software 
•Three  half-meter  Geonics  EM61  coils,  upper  and  lower 
•Six  channels  of  serial  data  acquired  at  10  Hz 

•1  Hz  DGPS  input  for  precise  EM6i  positioning  (GPS  antenna  located  over 
center  coil) 

6.1.5  Data  Processing  and  Interpretation 

On-site  data  processing  uses  a  Silicon  Graphics  Unix  workstation  with  100  megabytes  of 
memory  and  proprietary  X/Motif  software  to  combine  the  sensor  data  with  the  DGPS  data 
and  create  spatially  registered  data  images  of  the  surveyed  area.  Data  processing  involves 
time-correlating  and  subtracting  the  reference  magnetometer  data  from  the  vehicular 
magnetometer  data,  correcting  for  errors  in  the  navigation  and  heading  data,  individually 
calibrating  the  sensors,  and  interpolating  the  sensor  data  onto  a  10  cm  grid  for  visual 
display.  Observable  surface  features  are  surveyed  with  the  DGPS  to  create  a  file  of  GPS 
locations  that^can  be  overlaid  on  a  data  image  to  correlate  anomalies  with  trees,  buildings, 
manholes,  wells,  and  other  cultural  objects.  Data  are  exported  at  several  raw, 
preprocessed,  and  processed  stages  in  a  variety  of  file  formats  compatible  with  standard 
GIS,  signal  processing,  and  image  processing  software  packages. 

Magnetic  data  analysis  includes  visually  identifying  anomalous  areas  of  interest  in  the 
total  field  data,  and  using  a  three-dimensional  nonlinear  least  squares  curve  fit  to  a  model 
of  a  magnetic  dipole  to  extract  location,  size,  depth,  and  angular  parameters.  The  gradient 
and  a  gradient  model  is  then  examined  to  further  refine  and  resolve  compound 
objects.  In  the  case  of  electromagnetic  (EM61)  data,  depth  estimates  are  made  using  data 
from  the  lower  and  upper  coils.  Target  picks  from  the  magnetometer  and  EM  data  are 
spatially  correlated  to  produce  a  final  list. 

6.1.6  Quality  Assurance 

STOLS®  is  a  field  hardened,  ruggedized,  all  terrain,  multisensor  survey  system.  With 
over  4  years  of  deployment  experience  and  over  50  commercial  surveys  under  its  belt, 
major  design  reliability  issues  have  been  addressed.  The  system  is  fielded  with  a  set  of 
spares,  including  tools  and  sheltered  work  area  for  affecting  necessary  repairs.  System 
status  indicators  are  provided  to  the  operator  during  operations.  Data  are  downloaded 
several  times  per  day  during  survey  operations  and  immediately  displayed  and  pre- 
processed  to  vaUdate  system  performance  and  to  document  coverage  to  date.  Once 
validated,  routine  data  processing  is  initiated  on-site.  Any  suspect  data  is  reacquired,  if 
necessary.  Daily  coverage  maps  are  provided  so  that  progress  can  be  assessed,  monitored 
and  documented.  The  control  monuments  and  grid  locations  are  overlaid  on  sensor  image 
Hata  to  visually  confirm  that  the  DPGS  is  correctly  set  up  and  reporting  locations  in  true 
latitude  and  longitude. 
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6.2  Demonstration  Results 


6.2.1  Assumptions 

The  main  assumption  in  data  analysis  is  that  the  unexploded  ordnance  and  their  depths 
are  commensurate  with  the  descriptions  for  each  scenario. 

6.2.2  Site-Specific  Procedures 

GEO-CENTERS  surveyed  scenarios  1,  2  and  3  with  a  single  mob/demob.  Each  scenario 
consisted  of  10  acres  for  a  total  of  30  acres.  The  specific  scenarios  surveyed  are  “Serial 
Gunnery  Range,”  “Artillery  and  Mortar  Range,”  and  “Grenades  and  Submunitions 
Range.”  These  scenarios  were  chosen  to  evaluate  the  multisensor  technology  separately 
for  specific  ranges  by  ordnance  type.  All  three  scenarios  were  surveyed  with  the  sensors 
at  a  6"  height.  Scenarios  1  and  3  were  surveyed  first  with  east-west  traverse  lines  to 
maximize  coverage.  Scenario  2  was  surveyed  last  with  north-south  traverse  lines. 

6.2.3  Problems  Encountered 

A  magnetometer  sent  out  for  repair  did  not  arrive  when  expected,  necessitating  starting 
the  survey  using  a  spare  magnetometer  which  was  functionally  equivalent  but  housed  in  a 
larger  tube  which  needed  to  be  secured  to  the  sensor  platform.  The  correct,  repaired 
sensor  arrived  on  the  second  day.  The  sensor  platform  experienced  one  flat  tire.  Rainy 
weather  slowed  progress  on  the  third  day  but  did  not  cause  any  shutdowns.  Several  of  the 
heavily  treed  sections  indicated  on  the  topographic  map  were  impenetrable  by  the  vehicle 
and  went  unsurveyed. 

6.2.4  Discussion  of  Results 

As  required,  results  are  provided  in  a  Microsoft  Excel  spreadsheet  called  geocen3.xls 
based  on  the  Excel  workbook  data_ent.xls  distributed  with  the  statement  of  work.  For 
analysis,  the  survey  area  was  broken  up  into  three  separate  sites  reflecting  scenarios  1-3. 
During  analysis,  the  operator  recorded  a  confidence  flag  based  on  the  visual  strength  of 
signals  in  the  magnetometer  and  EM  data.  These  confidence  flags  are  written  into  the 
spreadsheet  under  the  comment  field.  Targets  are  commented  “high  confidence”  if  they 
had  very  strong  mag  or  EM  signals,  or  if  a  discemable  signal  was  present  on  both  the  mag 
and  EM  sensors.  Targets  that  were  high  confidence  and  also  close  enough  to  another 
target  to  be  considered  “compound”  objects  were  commented  as  such.  Targets  were 
commented  “medium  confidence”  if  a  nominal  visually  identifiable  signal  was  present  on 
either  sensor.  Targets  were  commented  “low  confidence”  if  there  were  some  signal  that 
might  be  geology  or  noise.  A  fence  line  runs  north-south  through  scenarios  2  and  3,  and 
east-west  through  pan  of  scenario  3,  and  targets  within  approximately  15  meters  of  the 
visual  center  of  these  lines  were  commented  “within  15  meters  of  probable  fence  line.” 
Scenario  2  contained  an  additional  anomaly  —  a  very  strong  magnetic  “streak”  coincident 
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with  a  gully-like  geologic  formation.  This  is  probably  an  extended  run  of  barbed  wire 
lying  in  a  gully.  The  endpoints  of  this  anomaly  were  logged  as  “along  the  probable 
fence,”  but  several  objects  along  this  streak  were  individually  discemable,  particularly 
with  the  gradiometers  and  EM61  sensors,  and  these  were  logged  as  high-confidence 
targets.  Other  faint  effects  of  geology  can  be  seen  in  the  magnetometer  data.  If  a  scenario 
included  large  objects  which  could  have  anomalies  similar  to  geology,  large  signals 
which  probably  are  geologic  were  commented  “low  confidence.”  If  the  scenario 
precluded  large  objects,  spatially  large  anomalies  that  were  possibly  geologic  were  not 
logged  as  targets. 


It  should  be  stressed  that  STOLS®  is  a  very  high  resolution,  very  low  noise,  multisensor 
survey  system,  and  will  detect  very  small  amounts  of  ferrous  and  non-ferrous  metal  left 
over  from  prior  use  of  the  survey  area. 


Scenario  1 

Scenario  2 

Scenario  3 

High  Confidence 

194 

168 

195 

Medium  Confidence 

86 

149 

132 

Low  ConfitJ^nce 

39 

52 

68 

Along  Probable  Fence 

none 

72 

53 

6.2.5  Raw  Data 

The  raw  data  enclosed  are  from  the  STOLS®  main  control  computer  on  the  vehicle.  Files 
with  the  extension  “.mag”  are  sensor  data  files  containing  vehicular  GPS  and  sensor  data. 
These  files  are  in  a  binary,  compressed  GEO-CENTERS  format,  “.mag”  files  whose 
names  begin  with  “jpg3"  contain  eight  channels  of  magnetometer  data  at  20  Hz  and  GPS 
data  from  the  antenna  over  the  center  magnetometer,  “.mag”  files  whose  names  begin 
with  “0004"  contain  six  channels  of  EM61  data  at  10  Hz  and  GPS  data  from  the  antenna 
over  the  center  EM  coil.  Files  with  the  extension  “.ref’  are  reference  magnetometer  files 
from  the  diurnal  variation  station.  These  raw  data  files  occupy  approximately  22 
megabytes. 

6.2.6  Processed  Data 

Due  to  the  binary,  compressed  nature  of  raw  STOLS®  data  files,  a  second  set  of  processed 
files  is  provided  in  ASCII  format.  Files  with  the  extension  “.dat”  are  ASCII,  whitespace 
delimited  files  containing  latitude,  longitude,  and  sensor  values  of  the  STOLS  data  files 
before  interpolation.  Latitude  and  longitude  are  in  decimal  degrees.  All  appropriate 
navigation,  heading,  reference  magnetometer,  and  sensor  calibration  corrections  have 
been  applied  to  the  sensor  data.  Files  on  Unix  machines  can  have  longer  names  than  files 
on  PCs,  so  the  names  of  the  “.dat”  files  have  been  shortened  to  archive  them  on  PC- 
compatible  media,  “.dat”  files  whose  names  begin  with  “m"  are  from  the  magnetometer 
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array.  ”.dat”  files  whose  names  begin  with  “e"  are  from  the  EM61  array.  These 
preprocessed  data  files  occupy  approximately  113  megabytes. 

6.2.7  Conclusions 

STOLS®  functioned  nearly  flawlessly,  acquiring  roughly  30  acres  of  data  over  scenarios  1 
through  3  in  roughly  24  survey  hours.  The  only  system  of  its  kind  in  the  world,  STOLS 
simultaneously  high-resolution,  GPS -integrated  acquires  total  field  magnetometer, 
gradiometer,  and  EMbldata. 

6.2.8  References  -  None. 


V 
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Geo-Centers  -  Combined  Statistics:  Scenarios  1,2,3 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Po-’ 

P  random 

Ordnance 

208 

194 

0.93 

0.116 

Nonordnance 

166 

133 

0.80 

Total 

374 

327 

Number  False  Alarms 

1005 

False  Alarm  Rate  (#/Hectare) 

81.80 

Feilse  Alarm  Ratio  (#/Ord.) 

5.18 

Probability  False  Alarms 

0.103 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation 

Position  (x,v)  1 

dx  -  easting  error 

-0.18 

0.37 

dv  -  northing  error 

0.24 

0.37 

Radial  error 

0.53 

0.30 

Depth  (z)  1 

dz  -  averaged  depth  error 

0.38 

0.3  .  ! 

Idzl®-  absolilte  depth  error 

0.48 

Characterization  Statistics  (TMA  Closest) 


1  #  Baseline  |  #  Detected  I  Pd"  |#  Correct  Pc“  I 

lAbilitv  to  Tvoe 

Ordnance 

208 

194 

0.93 

194 

1.00 

Nonordnance® 

167 

133 

0.80 

0 

0.00 

lAbilitv  to  Size  - 1 

Large 

14 

14 

1.00 

7 

0.50 

Medium 

39 

37 

0.95 

14 

0.38 

Small 

155 

143 

0.92 

115 

0.80 

lAbilitv  to  Classifv  — 1 

Bomb 

21 

21 

1.00 

21 

1.00 

Projectile 

42 

40 

0.95 

39 

0.98 

Mortar 

26 

23 

0.88 

0 

0.00 

Submunition 

97 

88 

0.91 

88 

1.00 

Rocket 

22 

22 

1.00 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
‘Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
‘TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 
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7.0  BLACKHAWK  GEOMETRICS  ^ 

Blackhawk  Geometries  demonstrated  from  October  23  through  29,  1996  at  the  16-hectare 
area  at  JPG.  Blackhawk  Geometries  also  participated  in  Phases  I  and  n  of  the  UXO  ATD 
program. 

7.1  TECHNOLOGY  DESCRIPTION 

The  Geometries  team  used  a  ground-based  multi-sensor  approach  (magnetics  and  time- 
domain  electromagnetics)  for  the  JPG  Phase  IH  demonstration.  These  two  methods 
showed  the  highest  probability  of  detection  (Pd)  of  UXO  in  the  JPG  Phase  n 
demonstration.  We  believe  that  additional  improvements  in  the  Pd  and  a  reduction  in  the 
false  alarm  rate  (FAR)  will  result  by  combining  the  two  methods. 

The  geophysical  equipment  used  in  the  Phase  IQ  demonstration  were: 

The  Geometries  G-858  MagMapper  Magnetometer. 

The  Geonics  EM61  (TDEM)  Metal  Detector. 

A  Th^  Geonics  EM61-3D  (TDEM)  System. 

The  Geometries  G-858  MagMapper™  portable  magnetometer/Gradiometer  was 
developed  to  be  a  rapid,  efficient  in-field  data  collection  and  map  processing  unit.  It  was 
designed  for  a  variety  of  detection  applications  such  as  buried  drums,  tanks,  etc. 
Geometries  and  AETC  have  also  developed  maximum-likelihood  dipole  pattern- 
matching  software  (MagAID™)  that  allows  the  user  to  analyze  magnetic  data  quickly  and 
interactively.  Blackhawk  Geometries  has  independently  developed  a  genetic-algorithm 
approach  to  magnetic-target  estimation  based  on  the  field  of  a  uniformly  magnetized 
rectangular  prism. 


^  Demonstartion  Summary  Report,  Phase  III  UXO  Detection,  Identification,  & 
Remediation,  Advanced  Technology  Demonstration,  Jefferson  Proving  Ground, 
Madison,  Indiana 
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The  Geonics  EM61  (IDEM  Metal  Detector)  was  developed  to  detect  buried  metal  such 
as  drums,  storage  fuel  tanks,  utility  lines  and  a  variety  of  metal  objects  buried  in  trenches 
and/or  landfiDs.  The  system  was  specifically  designed  to  mitigate  the  influence  of 
variations  in  soil  properties  and  ambient  EM  noise.  The  system  has  been  used  for 
numerous  UXO  surveys;  however,  the  design  is  not  optimized  for  detection  of  small 
metallic  targets,  such  as  submunitions.  The  prototype  Geonics  EM61-3D  was  designed 
for  UXO  detection  by  incorporating  a  more  powerful  transmitter  and  by  measuring  three 
orthogonal  components  of  the  induced  magnetic  field  in  twenty  time  channels  (the 
standard  EM-61  records  the  vertical  component  only  at  a  single  time  gate).  The  EM61- 
3D  can  be  operated  to  record  a  continuous  data  stream  (dynamic  mode)  or  at  discrete 
locations  (static  mode)  analogous  to  standard  EM  exploration  procedure.  In  contrast  to 
other  demonstrators  who  used  the  dynamic  mode,  we  operated  the  EM61-3D  almost 
exclusively  in  the  static  mode  in  order  to  ensure  high  signal  to  noise  ratio. 

7.1.1  FIELD  METHODS 

7.1.1.1  SCENARIO  2  (10-Acre  Artillery  &  Mortar  Range) 

Data  for  Sceqaiio  2  were  acquired  with  a  hand-pulled  dual  array  of  cesium-vapor 
magnetometers  and  two  hand-pulled  EM61  TDEM  metal  detectors 

The  magnetometers  were  mounted  on  a  rigid-wheeled  cart  with  a  horizontal  sensor 
separation  of  18  inches  and  sensor  height  of  18  inches  above  the  ground.  The  array  was 
operated  by  a  single  person.  Magnetic  data  were  taken  with  the  array  on  lines  three  feet 
apart  at  a  sample  rate  of  10  samples  per  second,  resulting  in  a  magnetic  reading  about 
every  six  inches  along  lines  spaced  by  18  inches.  A  base  station  magnetometer  was  used 
to  remove  diurnal  variations  from  the  magnetic  field  data. 

The  two  EM61s  were  operated  in  the  standard  wheel-cart  mode.  Data  were  taken  on  3  ft 
line  spacing  using  an  optical  wheel  encoder  to  trigger  the  measurements.  The  result  is  an 
EM61  measurement  about  every  7  inches  along  lines  spaced  by  36  inches. 

Survey  lines  for  all  instruments  ran  site  north  to  site  south,  parallel  to  survey  stakes 
established  by  PRC.  The  field  coordinate  system  therefore  used  an  x-axis  to  site  west  and 
y-axis  to  site  south.  The  x-position  is  therefore  the  line  number  and  the  y-position  the 
station  number.  Line/station  coordinates  were  converted  to  UTM  using  known  site 
monuments. 


Survey  control  was  established  by  taping  and  spray-painting  the  3-ft  survey  lines  at  100-ft 
along-line  intervals  established  by  the  survey  stakes.  Fluorescent  traffic  cones  at  these 
100-ft  intervals  provided  instrument  operators  a  clear  position  of  the  survey  line  and 
Mucial  checks  were  performed  at  each  100-ft  mark.  The  accuracy  of  surveys  positioned 
this  way  is  expected  to  be  ±1  ft  and  yet  provides  comparatively  rapid  coverage  even  in 
rough  or  tree-covered  areas.  Data  acquisition  rates  for  both  EM61s  and  magnetometers 
were  each  better  than  two  acres  per  day  per  unit.  In  typical  field  conditions,  average 
productivity  with  magnetometers  tend  to  be  higher  than  with  the  EM61 .  Real-time 
quality  assurance  for  the  magnetics  survey  was  achieved  by  monitoring  the  MagMapper 
control  unit  which  contains  a  field-data  display  unit.  For  the  EM61  surveys,  data  quality 
assurance  was  monitored  by  repeat  test  segments  off  the  grid.  For  both  magnetics  and 
EM61  surveys,  the  data  were  partially  processed  in  the  field  to  verify  data  quality  and  to 
monitor  total  coverage  on  a  daily  basis. 


There  are  no  specific  weather  limitations  for  the  use  of  the  G-858  MagMapper™  dual¬ 
sensor  magnetic  detection  system  or  the  EM61  metal  detector.  Geological  limitations  for 
magnetic  measurements  can  occur  in  volcanic  areas.  The  limitation  is  not  in  the  ability  to 
measure  the  magnetic  anomalies  but  in  the  ability  to  separate  geologic  noise  from  targets 
of  interest.  There  are  minimal  cultural  limitations  since  both  sensors  are  not  affected  by 
power  lines  or  strong  RF  signals.  The  magnetic  method  is  only  effective  for  ferrous 
UXO,  so  that  non-ferrous  UXO  will  not  be  found  with  the  magnetometer.  TDEM  is 
effective  for  non-ferrous  and  ferrous  UXO  and  can  be  used  in  geologic  environments  that 
have  volcanic  materials.  The  biggest  limitation  for  existing  TDEM  systems  is  their 
limited  detection  range  (less  than  three  meters  depth).  TDEM  systems  and  magnetometer 
systems  are  complimentary.  They  each  measure  different  physical  parameters  of  the 
UXO.  Together  they  have  the  potential  to  better  locate,  discriminate,  and  characterize. 

7.1.1.2  SCENARIO  4  (20-Target  Interrogation  &  Burial  Area) 

Data  for  Scenario  4  were  acquired  with  magnetometers,  EM61s,  and  the  EM61-3D.  The 
magnetometer  cart  was  modified  to  include  two  additional  sensors  in  a  second  rack  18” 
above  the  original  two  sensors;  the  resulting  quad-sensor  configuration  can  then  produce 
both  horizontal-  and  vertical-gradient  data.  The  EM61-3D  was  operated  in  station-by- 
station,  or  static,  mode.  EM61  procedures  were  identical  to  those  in  Scenario  2. 

Twenty  red-flagged  targets  in  Scenario  4  were  assigned  to  Geometries.  In  order  to 
improve  visual  contrast  with  the  orange  group  of  flags,  the  red  flags  were  replaced  with 
blue  ones  (with  the  approval  of  PRC  personnel).  The  flag  locations  were  surveyed  by 
taping  their  positions  from  the  marked  grid  comers.  The  modeled  target  locations  have 
small  corrections  to  the  flagged  positions  of  order  tens  of  centimeters;  the  final  target 
locations  were  then  converted  to  UTM  coordinates. 

A  30'  X  30'  grid  was  tape-surveyed  around  each  target  flag  using  local  coordinate  systems 
with  the  same  orientation  as  the  field  coordinate  system  used  for  Scenario  2. 
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Magnetometer  and  EM-61  line  and  station  spacings  also  follow  Scenario  2.  Station 
spacing  for  the  EM-61-3D  was  1  to  3  ft. 

5.9.1.2  DATA  PROCESSING,  INTERPRETATION,  AND  RESULTS 


The  magnetic  data  collected  in  Phase  HI  were  forwarded  to  AETC  for  processing.  The 
combined  magnetic  and  TDEM  data  sets  were  processed  by  Blackhawk  Geometries.  The 
final  interpretation  of  the  data  was  made  by  synthesizing  the  combined  interpretation  by 
AETC  and  Blackhawk  Geometries. 

7.1.2.1  SCENARIO  2  (10  Acre  Site,  Artillery  &  Mortar  Range) 

The  magnetics  data  taken  at  Scenario  2  were  processed  and  interpreted  by  a  workstation 
version  of  MagAid™.  This  software  corrects  for  diurnal  variations  (drift),  heading  error 
(orientation  of  sensor),  and  performs  data  quality  assurance  checks.  The  graphical, 
interactive  interface  allows  rapid  analysis,  which  is  important  for  Scenario  2  where  a 
large  number  of  targets  are  identified.  Anomalies  are  visually  selected  and  outlined;  the 
program  then  finds  the  best-fitting  magnetic  dipole  for  the  selected  data  and  reports 
goodness-of-§t.  The  fitted  parameters  are  the  three-dimensional  location  of  the  target, 
the  dipole  angles  (inclination  and  declination),  and  dipole  moment.  The  target 
magnetization  can  be  determined  from  the  magnitude  of  the  Earth’s  field  under  the 
assumption  that  magnetization  is  induced.  The  size  of  a  sphere  matching  the  best-fitting 
dipole  then  follows  from  the  magnetization  and  the  inferred  magnetic  moment. 
Parameters  for  accepted  fits  are  recorded  in  a  target  list;  these  target  lists  were  used  to 
help  prepare  the  target  detection  sheets. 

EM61  data  were  used  mainly  for  target  identification.  However,  target  depths  and  sizes 
can  be  estimated  using  programs  developed  at  both  AETC  and  Blackhawk  Geometries 
incorporating  the  time-domain  response  of  a  ferrous  sphere.  Due  to  the  large  number  of 
targets  in  Scenario  2,  quantitative  EM61  interpretation  was  applied  principally  at 
Scenario  4;  the  methodology  is  described  in  that  section  below.  EM61  processing  is 
minimal,  involving  only  a  regional  bias  correction. 

The  combined  data  sets  (magnetics  and  TDEM)  for  Scenario  2  were  analyzed  in  two 
ways: 

1)  In  a  conventional  manner  in  which  magnetic  anomalies  and  TDEM  anomalies  are 
separately  located  and  characterized  by  the  methods  described  above.  Magnetic  and 
EM  anomalies  that  coincide  were  evaluated  by  using  histogram  equalization  analysis 
to  enhance  anomaly  contrast  in  a  given  image.  This  technique  is  useful  for 
magnetometer  and  EM61  data  sets  since  limited  contrasts  are  sometimes  observed  in 
these  data.  This  process  is  conducted  on  PCs  using  commercially  available  software. 
Color  grid  maps  of  the  magnetics  and  EM61  data  taken  over  Scenario  2  are  given  in 
Appendix  C  (Figure  1). 


B.7-5 


2)  By  using  an  advanced  sensor  fusion  technique.  Principal  Component  Analysis,  (PCA) 
to  statistically  recombine  the  magnetometer  and  TDEM  data  sets  into  images  of 
decreasing  variance  to  facilitate  detection  and  classification  of  anomalies.  The 
geophysical  anomalies  observed  in  these  images  can  be  identified  and  delineated 
more  robustly  compared  to  the  original  images.  The  PCA  code  is  proprietary  to 
Blackhawk  Geometries  and  is  run  on  a  workstation.  A  color  grid  map  of  the  second 
principle  component  for  the  data  taken  over  Scenario  2  is  given  in  Appendix  C 
(Figure  2).  The  PCA  clearly  outhnes  the  areas  of  coincident  magnetic  and  EM61 
anomalies,  and  reduces  ambiguities  caused  by  noise  in  the  individual  data  sets.  A  set 
of  overlays  of  the  targets  derived  from  the  PCA  for  the  Scenario  2  area  is  given  in 
Appendix  C  (Figure  3a-3d). 


In  Scenario  2,  UXO  target  locations  were  picked  by  several  methods; 

Manual  selection  based  on  magnetics  only. 

Manual  selection  based  on  EM61  only. 

AA-Utomatic  selection  of  combined  data  sets  with  Principal  Component 
Analysis  (PCA). 

When  anomalies  existed  in  both  magnetic  and  EM61  data,  a  comparison  of  manual 
selection  and  automatic  selection  with  PCA  showed  an  accurate  correspondence.  This 
comparison  gives  confidence  in  PCA  for  further  work.  Several  other  targets  which  were 
detected  by  only  one  method  (i.e.,  deep  targets  detected  by  magnetics,  non-ferrous  targets 
detected  by  EM61)  were  classified  by  an  operator  using  method  1  (above).  Examples  of 
deep  targets  detected  only  with  magnetics  are  target  numbers  158, 228,  and  239. 
Examples  of  non-ferrous  targets  detected  only  with  the  EM61  are  target  numbers  224- 
227.  Target  parameters  (type,  class,  depth,  etc.)  were  mainly  derived  from  the  magnetic 
data,  except  where  the  magnetic  data  were  ambiguous  or  targets  were  non-ferrous.  Non- 
ferrous  targets  detected  with  the  EM61  were  classified  as  ordnance,  but  do  not  have 
ferromagnetic  mass.  Thus,  the  weight  (requested  in  ferrous  material)  for  these  targets 
was  listed  as  “none.”  An  additional  column  “NF  Weight”  (Non-ferrous  weight)  was 
added  to  the  detection  sheet  to  accommodate  these  targets.  Obvious  non-ordnance 
anomalies  such  as  the  buried  fence  line  (north-south  linear  anomaly  on  west  side  of  site) 
were  not  reported  in  the  Detection  Sheet.  The  detection  sheet  and  floppy  disk  for 
Scenario  2  are  given  in  Appendix  D. 

7.1.2.2  SCENARIO  4  (20  Targets,  Interrogation  &  Burial  Area) 

The  magnetics  and  EM61  data  taken  in  Scenario  4  were  also  processed  in  the  same 
manner  as  in  Scenario  2.  EM61-3D  processing  involved  only  editing  of  field  errors. 

The  three  data  sets  were  interpreted  separately  and  the  results  weighed  qualitatively  for 
the  combined  interpretation.  Magnetic  theory  is  best  developed  at  present  and  so  the 
magnetometer  data  carry  the  greatest  weight  in  our  interpretation.  Two  independent 
methods  were  used  for  magnetic  interpretation.  The  first  was  AETC’s  dipole-fitting 
program  MagAid™,  described  above.  The  second  approach,  developed  at  Blackhawk 
Geometries,  is  more  general  but  broadly  similar.  This  technique  uses  the  analytical 
formulation  for  the  magnetic  field  from  a  uniformly  magnetized  rectangular  prism.  The 
method  therefore  has  the  potential  to  accurately  model  the  magnetic  field  of  elongated 
objects  (e.g.  UXO)  with  arbitrary  orientations  and  differing  induced  and  remnant  fields. 
For  simplicity  and  comparison  to  MagAid™,  however,  all  targets  were  modeled  as  cubes 
with  a  single  magnetizing  field.  A  second  procedural  difference  is  the  use  of  a  new  class 
of  global  optimization  methods,  genetic  algorithms,  to  solve  for  the  best-fitting  model 
parameters.  Both  magnetic  modeling  techniques  were  employed  and  compared.  In  all 
but  five  targets  (with  noisy  signatures),  there  was  good  agreement.  For  the  five  targets,  a 
compromise  in  target  parameters  between  the  two  modeling  procedures  was  used. 


The  time-domain  electromagnetic  response  of  a  spherical  target  to  the  EM61  can  also  be 
modeled,  and  depth  can  be  determined  from  either  the  anomaly  half-width  or  the  ratio  of 
upper-to-lower  receiver  coil  responses.  Given  the  target  depth  and  the  peak  voltage  on 
either  channel,  the  target  diameter  may  be  estimated.  These  models  consider  only  vortex 
currents  induced  in  a  conductive  target  and  do  not  account  for  the  so-called  "permeability 
currents"  caused  by  alignment  and  relaxation  of  magnetic  domains  in  a  ferromagnetic 
target.  EM61  depth  and  size  modeling  was  used  for  comparison,  more  weight  was  given 
to  the  magnetic  modeling. 

With  the  EM61  -  3D  a  large  data  set  is  recorded  per  station  (60  values),  and  the 
technology  for  effective  processing  is  not  yet  developed.  Moreover,  the  time  to  develop 
the  processing  capability  was  not  available  under  JPG  m.  We  evaluated  several  targets 
and  considered  visually  the  discrimination  potential  contained  in  the  data,  such  as  target 
orientation.  Some  of  these  visual  observations  have  been  included  on  the  data  sheets. 
Examples  of  visual  display  of  EM61  -  3D  data  are  shown  in  Figures  4  and  5  of  Appendix 
C.  Figure  4  is  data  acquired  on  the  Blackhawk  test  range  for  a  25  lbs  bomb,  showing 
dependence  on  target  orientation.  Figure  5  is  data  acquired  at  JPG  (Scenario  4,  Target 
1227)  showing  the  potential  for  better  positioning  and  orientation  information. 

For  either  magnetic  or  TDEM  data,  the  equivalent  (spherical)  volume  of  known  ordnance 
was  then  compared  to  the  inferred  target  volume  to  estimate  ordnance  type.  Three- 
dimensional  position  follows  directly  from  the  modeling  procedures.  Target  orientation 
was  estimated  principally  when  the  computed  magnetization  directions  differed  strongly 
from  the  natural  field,  under  the  assumption  that  magnetization  generally  follows  the  long 
axis  of  an  elongated  body.  Secondary  estimates  of  position  and  orientation  were  inferred 
from  the  EM-61 -3D. 

The  final  target  parameters  were  obtained  by  combining  the  results  from  all  of  the 
estimating  methods.  The  target  detection  sheet  and  floppy  disk  for  Scenario  4  data  are 
given  in  Appendix  D. 


»  CONCLUSIONS 


The  Geometries  team  used  a  multi-sensor  approach  (magnetics  and  EM61)  using  the  two 
sensors  which  showed  the  highest  probabiUty  of  detection  (Pd)  in  previous  JPG 
demonstrations.  The  survey  procedure  selected  was  ground-based  (man-carried)  using 
equipment  and  positioning  techniques  commonly  used  in  commercial  geophysical 
studies.  This  procedure  is  applicable  to  all  terrain  types,  and  resulted  in  rapid 
deployment,  no  breakdowns  or  lost  time  due  to  equipment  failures,  and  nearly  100% 
coverage  of  the  site. 

From  the  interpretation  of  both  magnetics  and  EM61,  we  concluded: 


B.7-8 


Greater  fhan  90  percent  of  the  targets  show  coincident  anomalies  in  both  methods. 
Deeply  buried  (greater  than  three  meters)  targets  were  only  detected  with 
magnetics. 


¥ 
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B.Non-ferrous  targets  were  only  detected  with  the  EM61. 

In  terms  of  operational  capability,  we  observed: 

Magnetic  surveys  are  more  portable  and  can  achieve  a  higher  productivity. 

Both  methods  are  influenced  by  non-ordnance  metallic  structures  (buried  fence 
line). 

Significant  new  developments  from  the  data  analysis  at  JPG,  Phase  HI  are: 

The  automatic  target  detection  with  principal  component  analysis  (PCA)  is 
accurate  when  coincident  anomalies  in  magnetic  and  EM61  data  sets  occur. 
PCA  requires  two  data  sets  and  we  have  already  proved  that  the  two  channel 
readings  of  the  EM61  also  can  be  used  to  perform  a  PCA. 

A  large  data  set  is  collected  with  the  EM61  -  3D  system.  There  was  no  time  to 
fuUy  analyze  the  data  set  in  the  time  frame  allotted.  Visual  observations  of 
graphical  displays  of  the  data  indicate  a  high  potential  for  target 
discrimination. 
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Geometries  -  Combined  Statistics:  Scenario  2  oniy 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

67 

60 

0.90  1  0.063 

Nonordnance 

50 

42 

0.84 

Total 

117 

102 

Number  False  Alarms 

180 

False  Alarm  Rate  (#/Hectare) 

38.44 

False  Alarm  Ratio  (#/Ord.) 

3.00 

Probability  False  Alarms 

0.048 

Locaiizatlon  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation 

Position  (x,v)  1 

dx  -  eastinq  error 

-0.01 

0.23 

dv  -  northing  error 

0.21 

0.38 

Radial  error 

0.39 

0.30 

iDepth  (z) _ 

dz  -  averaged  depth  error 

0.12 

0.29  II 

Idzl'-  absolute  depth  error 

0.32 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  |  #  Detected  |  Pd“  |#  Correct)  Pc"  I 

f  \  nil  i'll  li'i '  — — — — 

Ordnance 

67 

60 

0.90 

59 

0.98 

Nonordnance 

50 

42 

0.84 

0 

0.00 

lAbilitv  to  Size  1 

3 

3 

1.00 

1 

0.33 

Medium 

31 

29 

0.94 

12 

0.41 

Small 

33 

28 

0.85 

26 

0.93 

lAbilitv  to  Classify  1 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

39 

0.95 

11 

0.28 

Mortar 

26 

21 

0.81 

19 

0.90 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  '  Target  Matching  Aigorithm 

”  Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
NA  •  not  applicable 
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Depth  (meters) 


JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator;  Geometries 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 
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8.0  GEOPHEX* 

Geophex  demonstrated  from  October  23  through  27,  1996  at  the  16-hectare  area  at  JPG. 
Geophex  also  participated  in  Phase  11  of  the  UXO  ATD  program. 

8.1  TECHNOLOGY  DESCRIPTION 


During  the  JPG  demonstration  we  used  a  combination  of  the  GEM-3  (Geophex 
ElectroMagnetic  instrument,  version  3;  Figure  1)  and  a  commercially  available  cesium- 
vapor  magnetic  gradiometer.  The  GEM-3  is  a  frequency  domain,  electromagnetic 
induction  sensor  developed  by  Geophex,  Ltd.  It  is  a  unique,  lightweight  (total  weight  of 
10  lb.),  hand-held  instrument.  The  GEM-3  transmits  a  primary  electromagnetic  field  that 
induces  a  secondary  current  in  electrically  conductive  targets.  It  then  senses  the 
secondary  magnetic  field  produced  by  these  induced  currents.  The  primary  transmitted 
field  is  made  up  of  a  multi-frequency  composite  waveform,  allowing  different  penetration 
depths  in  the  earth,  depending  on  conductivity  structure.  The  GEM-3  consists  of  a 
transmitter  and  receiver  coil  assembly,  a  digitally  controlled  transmitter,  a  low-noise 
analog  receiver,  an  analog-to-digital  converter,  and  a  custom  digital  signal  processing 
unit.  The  jnagnetometer  used  was  a  Geometries  G-858,  cesium-vapor  magnetic 
gradiometer.  We  feel  that  when  combined,  the  use  of  magnetic  and  electromagnetic 
sensors  provides  a  more  effective  technique  for  UXO  surveys  than  either  technology 
alone. 
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*  GEOPHEX,  Final  Demonstration  Report  on  Geophex’s  Technology 
Demonstration,  December  1996 
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8.1.1  Assumptions  Made  Prior  to  Demonstration 

We  assumed  that  targets  of  interest  in  the  JPG  demonstration  could  be  either  magnetic 
and  non-magnetic  conductors.  For  this  reason  we  elected  to  use  total  magnetic  field, 
vertical  magnetic  field  gradient,  and  electromagnetic  induction  instruments.  No  other 
assumptions  were  made  prior  to  the  demonstration. 

8.1.2  Procedures  Used  to  Conduct  Demonstration 

Data  were  collected  in  a  grid  pattern  using  the  grid  coordinate  system  already  in  place  at 
JPG.  Grid  lines  were  laid  out  and  each  instrument  traversed  these  lines  while  collecting 
fjata  All  instruments  used  by  Geophex  personnel  were  man-portable.  Data  point 
locations  were  determined  via  a  dead  reckoning  procedure. 

8.1.3  Raw  Data  Obtained 

The  raw  data  collected  during  surveys  conducted  at  JPG  consisted  of  magnetic  (total  field 
and  vertical  gradient)  and  electromagnetic  (GEM-3)  readings.  Secondary  electromagnetic 
field  data  were  presented  as  parts  per  million  of  the  primary  transmitted  field.  The  20 
acres  includid  in  scenarios  1  and  2  were  surveyed  with  both  magnetic  and 
electromagnetic  sensors  in  less  than  the  40  hours  allotted.  This  time  period  includes  time 
spent  re-acquiring  data  over  areas  where  preliminary  data  analyses  indicated  less  than 
optimal  data  quality.  Additionally,  the  entire  20  acre  site  was  surveyed  twice,  with  both 
instruments,  once  with  traverse  lines  in  the  east-west  direction,  and  again  with  lines  in  a 
north-south  orientation. 

8.1.4  Data  Processing 

The  raw  data  were  plotted  in  contour  maps  from  which  we  determined  the  target  location 
and  other  attributes. 

The  target  data  information  include  the  target  identification  number,  spatial  coordinates 
(northing  and  easting),  depth  (m),  type  (ordnance  or  non-ordnance),  weight  (light, 
moderate,  or  heavy),  and  size  (small,  medium,  or  large). 

8.1.5  Technology  Capabilities  and  Limitations 

GEM  type  instruments  have  been  successfully  used  by  Geophex  personnel  since  1992  to 
map  targets  as  varied  as  contaminant  plumes,  buried  landfills,  and  underground  storage 
tanks.  Testing  by  Geophex  staff  members  has  shown  the  GEM-family  of  instruments  to 
be  effective  in  locating  many  different  kinds  of  buried  targets  and  debris.  These  include 
tanks,  wells,  pipelines,  barrels,  and  the  site  of  a  former  underground  nuclear  detonation. 
Since  the  GEM-3  is  a  hand-held  instrument,  a  major  limitation  is  the  ability  and 
endurance  of  the  operator.  As  with  ail  geophysical  techniques,  magnetic  and 
electromagnetic  data  suffer  from  non-uniqueness.  That  is,  many  different  targets  arc 
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capable  of  producing  the  same  magnetic  or  electromagnetic  signature.  This  is  probably 
the  major  limitation  of  these  technologies. 

During  the  course  of  the  demonstration  we  did  experience  some  equipment  problems  with 
the  magnetometer,  but  this  did  not  materially  effect  our  data  acquisition  efforts.  We 
experienced  no  equipment  malfunctions  with  the  GEM-3. 

Considering  the  results  of  our  demonstration  at  JPG  11  and  ancillary  experience,  we  feel 
the  combination  of  magnetic  and  GEM-3  electromagnetic  sensors  yield  a  powerful 
combination  of  data  sets  for  location  of  UXO  targets. 


V 
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Geophex  -  Combined  Statistics:  Scenarios  1,2 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Pd^  P  random 

Ordnance 

110 

85 

0.77  1  0.053  1 

Nonordnance 

127 

89 

0.70 

Total 

237 

174 

Number  False  Alarms 

264 

False  Alarm  Rate  (#/Hectare) 

32.44 

False  Alarm  Ratio  (#/Ord.) 

3.11 

Probability  False  Alarms 

0.041 

Locaiization  Statistics  (TMA  Closest)  (in  meters) 


_ _ _ 

Mean 

Std  Deviation 

Position  (x,y) 

dx  -  easting  error 

-0.06 

0.45 

dv  -  northing  error 

0.17 

0.39 

Radial  error 

0.51 

0.34 

[Depth  (z) _ 

dz  -  averaged  depth  error 

-0.05 

0.27  __|! 

Idzl'-  absolute  depth  error 

0.26 

Characterization  Statistics  (TMA  Closest) 


1  #  Baseline  |  #  Detected 

Po“  1#  Correct  Pc“  I 

lAbilitv  to  Tvoe  - 1 

Ordnance 

110 

85 

0.77 

85 

1.00 

Nonordnance® 

128 

89 

0.70 

2 

0.02 

lAbilitv  to  Size  1 

Large 

14 

14 

1.00 

5 

0.36 

Medium 

38 

27 

0.71 

16 

0.59 

Small 

58 

44 

0.76 

22 

0.50 

lAbilitv  to  Classify  J 

Bomb 

21 

21 

1.00 

0 

0.00 

Projectile 

41 

30 

0.73 

0 

0.00 

Mortar 

26 

15 

0.58 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

19 

0.86 

0 

0.00 

Notes:  *  Target  Matching  Aigorithm 

^  Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
^MA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 
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JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator:  Geophex 

Small  Medium  Large 


Principal  Diameter  (millimeters) 


■  Target  Detected 
□  Target  Not  Detected 
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9.0  GEOPOTENTIAL  ^ 


GeoPotential  demonstrated  from  September  3  through  6,  1996  at  the  16-hectare  area  at 
JPG.  GeoPotential  also  participated  in  Phase  II  of  the  UXO  ATD  program. 


9.1  TECHNOLOGY  DESCRUTION 

9.1.1  Sensor  System  and  Transport  Mode 

The  AQUA-TRONICS  A6  TRACER  is  manufactured  by  AQUA-TRONICS  in  Redmond, 
Oregon.  It  is  a  man-portable  elctromagnetic  induction  instrument  operating  at  1 17  kHz. 
The  instrument  consists  of  transmitting  and  receiving  antennas  separated  by  a  1.22  meter 
handle.  Subsurface  features  such  as  UXO  wUl  cause  variations  in  the  shape  of  the 
transmitted  wave  which  will  be  detected  as  voltage  variations  by  the  receiving  antenna. 


^  (GeoPotential ,  Phase  HI  Unexploded  Ordnance  Advanced  Technology 
Demonstration  Program  -  Survey  Data  Analysis  Report  -  October  5, 1996). 


9.1.2  Recommended  Applications 


The  A6  TRACER  was  developed  over  20  years  ago  to  detect  subsurface  utilities  such  as 
conductive  pipes  and  power  cables  and  is  an  excellent  tool  for  these  applications. 
GeoPotential  has  worked  with  AQUA-TRONICS  over  the  last  several  years  to  adapt  the 
A6  TRACER  for  the  applications  such  as  locating  buried  waste  drums,  underground 
storage  tanks  and  other  near  surface  objects  and  is  now  used  routinely  by  GeoPotential  for 
these  applications. 

The  A6  TRACER  is  limited  to  near  surface  features.  Variations  in  soil  moisture  and  soil 
types  can  cause  voltage  changes  which  can  create  false  positive  anomalies.  Surface 
features  such  as  overhead  power  lines,  vehicles,  buildings,  etc.  can  create  interference 
during  a  survey. 

9.1.3  Logistics  Requirements 

The  A6  TRACER  is  man  portable  and  can  be  operated  in  any  areas  where  the  instrument 
can  be  carried.  It  is  necessary  to  maintain  a  level  orientation  between  the  antennas  since 
the  instrument  is  sensitive  to  tilting. 

9.1.4  Data  Acquisition 

For  the  UXO  PHASE  IH  demonstration  two  A6  operators  used  the  A6  TRACER  in  the 
search  mode  utilizing  the  audible  output  to  locate  possible  UXO.  These  targets  were 
flagged  on  the  site.  Next  a  6X6  meter  grid  was  laid  out  over  the  target  and  5  east-west 
profiles  with  a  1  meter  profile  spacing  and  1  north-south  profile  were  recorded  with  the 
A6  TRACER  in  the  mapping  mode.  Data  was  recorded  on  a  OMNIDATA  PR02000  data 
logger  at  a  1  second  sampling  interval. 

The  UTM  COORDINATES  of  the  targets  were  measured  using  a  SOKKIA  GIRIOOO 
GLOBAL  POSITIONING  SYSTEM  in  the  differential  mode. 

9.1.5  Data  Processing  and  Interpretation 

A6  data  for  each  target  were  downloaded  to  a  LAPTOP  computer.  The  5  east-west 
profiles  were  gridded  and  contoured  to  form  the  EM  CONTOUR  maps  provided  for  each 
target.  In  addition  the  north-south  profile  was  plotted  along  with  the  contour  map  for  each 
target.  North-south  profiles  cross  the  middle  of  the  grid  at  meters  east  =  3. 

The  contour  maps  were  evaluated  to  determine  the  azimuth,  declination,  depth  and  size 
for  each  target.  Anomaly  shape  and  gradient  was  used  to  interpret  target  azimuth  and 
declination.  Anomaly  amplitude  and  wavelength  was  used  to  inerpret  target  depth  and 
size.  The  north-south  profile  was  used  to  identify  linear  targets  which  had  a  east-west 
orientation  since  this  sort  of  target  will  not  produce  a  significant  anomaly  along  the  east- 
west  profiles.  This  is  caused  by  the  fact  that  a  linear  target  produces  a  maximum  A6 
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TRACER  anomaly  when  oriented  perpendicular  to  the  direction  of  traverse  and  a 
TTiinimiim  anomaly  when  oriented  parallel  to  the  direction  of  traverse. 

9.1.6  Quality  Assurance 

Repeats  of  targets  with  the  GPS  system  provided  repeatability  within  .4  to  .6  meters 
which  was  considered  of  sufficient  quality  for  the  statistical  requirements  of  the  UXO 
program. 

A6  TRACER  quality  was  tested  by  repeating  several  targets  to  assure  repeatability  of 
results. 

9.2  DEMONSTRATION  RESULTS 

9.2.1  Assumptions 

Targets  which  produced  EM  ANOMALIES  which  were  of  the  size  and  shape  compatible 
with  UXO  were  assumed  to  be  caused  by  UXO. 

9.2.2  Problems  Encountered 

V 

Along  some  of  the  A6  TRACER  PROFILES  and  corresponding  contour  maps  dc  level 
shifts  and  ramps  were  found  as  noise  in  the  A6  data.  In  most  cases  it  was  possible  to 
resolve  this  noise  from  the  target  anomaly. 

9.2.3  Discussion  of  Results 

The  results  demonstrated  that  of  the  147  targets  detected  in  the  survey;  93  occurred  in 
SCENARIO  1, 22  in  SCENARIO  2  and  32  in  SCENARIO  3  (GeoPotential  did  not  survey 
SCENARIO  4).  It  is  most  probable  the  larger  targets  in  SCENARIO  1  made  them  more 
receptive  to  detection  with  the  A6  TRACER.  Utilizing  the  TRACER  in  the  search  mode, 
where  the  audible  output  of  the  TRACER  is  used  for  target  detection,  makes  detection  of 
smaller  targets  difficult. 

If  the  noise  encountered  in  the  A6  mapping  mode  (dc  level  shifts  and  ramping)  can  be 
diagnosed  and  eliminated  then  the  A6  TRACER  could  be  used  to  map  large  areas  and 
identify  smaller  target  sizes.  GeoPotential  and  AQUA-TRONICS  are  currently 
researching  the  noise  problems  encountered  during  mapping  mode. 

9.2.4  Conclusions 

The  A6  TRACER  is  an  effective  instrument  to  detect  the  types  of  TARGETS  in 
SCENARIO  I  when  used  in  a  combination  SEARCH  and  MAPPING  modes. 


If  noise  problems  encountered  in  the  MAPPING  MODE  can  be  solved  then  the  A6 
TRACER  should  be  an  effective  instrument  for  detecting  smaller  UXO  such  as 
encountered  in  SCENARIO  2  and  SCENARIO  3. 


GeoPotential  -  Combined  Statistics:  Scenarios  1 ,2,3 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Pd“ 

K2SS9i 

Ordnance 

208 

13 

0.06 

0.013 

Nonordnance 

166 

15 

0.09 

Total 

374 

28 

Number  False  Alarms 

111 

False  Alarm  Rate  (#/Hectare) 

9.04 

False  Alarm  Ratio  (#/Ord.) 

8.54 

Probabilitv  False  Alarms 

0.011 

Locaiization  Statistics  (TMA  Closest)  (In  meters) 


Mean 

Std  Deviation  | 

Position  fx.v) 

dx  -  eastinq  error 

0.12 

0.61 

dv  -  northing  error 

-0.19 

0.6 

Radial  error 

0.73 

0.49 

Depth  (z) _ 

dz  -  averaged  depth  error 

0.36 

0.6  ! 

IdzF-  absolute  depth  error 

0.69 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Pd“ 

#  Correct 

lAbilitv  to  Tvoe  - ^ - 1 

Ordnance 

208 

13 

0.06 

13 

1.00 

Nonordnance® 

167 

15 

0.09 

1 

0.07 

Abllitv  to  Size 

Large 

14 

3 

0.21 

1 

0.33 

Medium 

39 

8 

0.21 

7 

0.88 

Small 

155 

2 

0.01 

0  n 

0.00 

lAbilitv  to  Ciassifv  - 1 

Bomb 

21 

3 

0.14 

0 

0.00 

Projectile 

42 

2 

0.05 

0 

0.00 

Mortar 

26 

1 

0.04 

0 

0.00 

Submunition 

97 

0 

0.00 

0 

NA 

Rocket 

22 

7 

0.32 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
^MA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 
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Depth  (meters) 


10.0  GEOPHYSICAL  RESEARCH  INSTITUTE^® 

The  Geophysical  Research  Institute  (GRI)  demonstrated  from  October  9  through  13, 1996 
at  the  16-hectare  area  at  JPG.  GRI  was  a  first  time  participant  in  the  UXO  ATD  program. 

10.1  TECHNOLOGY  DESCRIPTION 

The  TM-4  &  TM-48  employ  the  same  instrumentation  system  interfaced  with  either  total 
field  magnetic  or  multi-period,  transient  EM  sensors  respectively. 


10.1.1  TM-4  Magnetometer  System 

The  TM-4  magnetometer  is  a  turnkey  data  acquisition,  processing,  interpretation  and 
documentation  package  designed  to  efficiently  detect  and  locate  ferrous  items.  It  was 
demonstrated  as  a  two-person  operation,  with  four  sensors  (Geometries  G822AS) 
recording  simultaneously  from  parallel  survey  lines.  Total  field  measurements  are 
automatically  recorded  at  0. 1  metre  intervals  irrespective  of  traverse  speed. 


The  Geophysical  Research  Institute,  Final  Report  for  UXO  Detection, 
Identification  and  Remediation  Technology  Demonstration  -1996, 13  November 
1996 


10.1.2  TM-4e  Electromagnetic  System 

The  TM-4e  is  a  turnkey  system  designed  to  efficiently  detect  and  locate  both  ferrous  and 
non-ferrous  metallic  sources.  It  is  immune  to  interference  from  magnetic  minerals  in  the 
ground,  is  able  to  detect  both  large  and  very  small  (including  the  detonator  of  a  plastic  AP 
mine)  items  and  able  to  resolve  between  items  that  may  be  close  to  each  other.  The  TM- 
4£  shares  the  proven  TM-4  data  acquisition  hardware,  and  data  processing,  interpretation 
and  documentation  software  package.The  TM-4e  demonstrated  used  asingle,  18  inch 
diameter  coil  sensor  (Minelab  F1A4).  It  was  operated  by  one  person.  Multi-period, 
transient  EM  measurements  were  automatically  recorded  at  regular  0.05  metre  intervals 
regardless  of  survey  speed. 


10.1.3  Recommended  Applications  and  Technology  Limitations 

10.1.3.1  TM-4  Magnetometer  System 

In  EOD  applications  magnetometers  are  suitable  only  for  detecting  ferrous  items.  The 
TM-4  is  use^to  greatest  advantage  when  its  survey  specification  has  been  optimised  for 
targets  deeper  than  0.3  metres  as  a  shallow  search  is  most  efficiently  conducted  using  the 
complementary  TM-4e.  Multiple  magnetic  sensors  may  be  used  in  ail  terrain  conditions 
that  are  accessible  on  foot  provided  the  density  of  trees  is  sufficiently  sparse  as  to  permit 
the  sensor  array  to  pass  through. 

10.1.3.2  TM-48  Multi-period,  Time  Domain  EM  System 

The  TM-4e  is  suitable  for  detecting  all  metals,  ferrous  and  non-ferrous.  It  may  be 
hand-held  or  vehicle-towed.  The  TM-4e  is  particularly  suited  to  locating  UXO  in 
geological  environments  that  contain  magnetic  minerals  near  the  surface.  Such  situations 
occur  in  magnetite  rich  volcanic  basalts  and  in  terrains  containing  laterite.  Ability  to 
detect  very  small,  near  surface  items  and  to  resolve  between  close  targets  makes  the  TM- 
4e  an  ideal  complement  to  the  deep  search  performance  of  the  TM-4. 

10.1.4  Logistics  requirements 

The  TM-4  and  TM-4e  systems  are  designed  to  be  readily  transportable  and  operational 
with  a  minimum  of  logistical  support.  Both  instruments  pack  into  cases  permitted  for 
airline  travel  as  personal  baggage.  Battery  charging  power  requirements  can  be  met  from 
automotive  12  volt  supply  if  mains  is  not  available. 


10.1.5  Data  Acquisition 

The  quad  sensor  TM-4  magnetic  data  to  0.01  nT  resolution  were  recorded  at  a  sensor 
elevation  of  0.65  metres,  with  0. 1  metres  sample  interval  along  lines  and  0.5  metre 
separation  between  lines.  A  base-station  (Geometries  G856)  magnetometer  was  used 
recording  to  0. 1  nT,  each  five  seconds. 
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At  each  controlled  site,  TM-4e  electromagnetic  data  were  recorded  at  a  mean  sensor 
elevation  of  0.1  metres,  sample  interval  of  0.05  metres  and  survey  line  spacing  of  0.5 
metres. 

In  both  the  TM-4  and  TM-48  applications,  position  control  along  survey  lines  was 
determined  by  cotton  thread  odometer  and  control  lines  of  known  separation  located  at 
grid  markers  provided  by  PRC.  Across  line  control  was  achieved  by  the  use  of  a  line  of 
visual  markers  located  at  survey  chains  laid  along  the  control  lines.  The  use  of  DGPS  is 
optional  with  both  TM-4  and  TM-4E  systems. 

10.1.6  Data  Processing  and  Interpretation 

Data  processing  was  performed  on  a  standard  IBM-compatible  PC  (minimum  486  with  8 
Mb  RAM  and  100  Mb  HDD).  In  the  main,  GRI  employed  Toshiba  410  lap-tops  with 
Pentium  90  processor,  16  Mb  RAM,  720  Mb  HDD, 

10.1.6.1  TM>4  Magnetic  Data 

The  TM-4  positional  data  was  corrected  by  using  the  control  line  information  recorded 
during  data  Requisition.  Compensation  was  also  performed  to  remove  the  temporal 
magnetic  disturbances  recorded  at  the  base-station  magnetometer.  This  data  were  stored 
as  a  raw,  positioned  data  file.  The  data  were  then  heading  corrected  for  sensor  orientation 
and  high  pass  filtered  to  remove  interference  from  geological  sources  below  10  metres. 
Next,  data  validation  and  QA  procedures  were  performed. 

The  data  interpretation  process  involved  computer-aided,  3-D  modelling  of  each  magnetic 
anomaly  and  comparison  with  a  UXO  knowledgebase.  The  output  of  this  interpretation 
process  was  a  database  file  “GTLMG.XLS”  containing  position,  mass,  size,  depth  and 
orientation  of  each  target  recognised. 

10.1.6.2  TM-4e  Electromagnetic  Data 

The  TM-4e  positional  data  was  corrected  by  using  the  control  line  information  recorded 
during  data  acquisition.  This  data  were  then  stored  as  a  raw  positioned  data  file.  Next, 
data  validation  and  QA  procedures  were  performed.  The  data  interpretation  process 
involved  manual  and  automatic,  computer-aided  identification  of  each  electromagnetic 
anomaly  followed  by  interpretation  using  a  UXO  knowledgebase.  The  output  of  this 
interpretation  process  was  a  database  file  “GTLEM.XLS”. 

10.1.6.3  Integrated  TM>4  and  TM-48  Data 

The  database  files  GTLMG.XLS  and  GTLEM.XLS  were  then  integrated  in  order  to: 

•  take  greatest  advantage  of  the  optimised  deep  detection  capability  of  the  TM-4; 

•  detect  non-ferrous  as  well  as  ferrous  targets; 

•  discriminate  against  magnetic  false  negatives  using  immunity  of  the  TM-4e  to 
mineralised  soil;  and 

•  utilise  the  TM-48  to  resolve  between  small  near  surface  items. 
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The  database  file  containing  the  integrated  interpretation  has  been  named  GTLCB.XLS. 

10.1.7  Quality  Assurance 

Quality  assurance  procedures  were  applied  to  each  phase  of  the  operation.  These 
procedures  included: 

•  instrument  calibration  checks  with  known  response  source  at  each  power-up  and 
power-down; 

•  continuous,  in-built,  instrumentation  self  diagnostics  with  audio  and  visual  alerts; 

•  routine  odometer  cahbration  check  during  data  pre-processing; 

•  routine  cross-correlation  positional  accuracy  check  during  data  processing; 

•  routine  image  cross-correlation  data  validation  check  during  pre-processing; 

•  routine  duplication  of  interpretation  modelling;  and 

•  routine  data  back-up  during  all  stages. 


10.2  Demonstration  Results 

10.2.1  Assumptions 

All  electrorriSignetic  responses  exceeding  the  system  noise  threshold  were  assumed  to 
have  as  their  source  a  potentially  hazardous  item  of  metallic  UXO  unless  the  dimensions 
of  the  interpreted  source  were  inconsistent  with  those  of  UXO  or  a  burial  pit  potentially 
containing  UXO. 

10.2.2  Site-Specific  Procedures 

The  occurrence  of  standing  and  fallen  vegetation  and  erosion  gullies  dictated  that 
hand-carried  operation  using  the  in-built  odometer  and  control  lines  (rather  than  the 
optional  DGPS)  would  be  the  most  efficient  and  cost-effective  procedure  at  this  site. 

10.2.3  Problems  Encountered 

One  magnetic  sensor  incurred  minor  damage  during  transit  and  as  a  consequence  required 
an  abnormal  warm-up  period  before  becoming  fully  functional. 

Grid  markers  at  the  four  comers  of  the  entire  demonstration  area  had  been  accurately 
located  by  surveying  prior  to  the  UXO  data  acquisition  survey.  However,  grid  markers 
between  these  points  were  not  accurately  located.  As  accurately  located  survey  control 
lines  rather  than  control  points  are  best  used  for  survey  control,  reliance  on  the 
intermediate  markers  would  give  a  potential  source  of  uncertainty  in  data  positioning. 
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10.2.4  Discussion  of  Results 

The  attached  tables  summarize  the  results  at  each  of  the  three  sites  investigated. 


TABLE  1 

Aerial  Gunnery  Range 


Total  area  surveyed: 

3.45  hectares 

Magnetic  sensor  elevation: 

0.65  m 

Magnetic  sample  interval  along  traverses: 

0.1  m 

Magnetic  sample  interval  across  traverses: 

0.5  m 

Number  of  magnetic  data  measurements: 

691,740 

Survey  duration: 

10.5  hours 

Magnetic  measurement  system  noise: 

0.2  nT 

Magnetic  interpretation  noise  threshold: 

InT 

Number  of  interpreted  ferrous  items: 

880 

Electromagnetic  sensor  elevation: 

0.05  m 

Electromagnetic  sample  interval  along  traverses: 

0.05  m 

Electromagnetic  traverse  width: 

0.5  m 

Number  of  electromagnetic  data  measurements: 

1,383,480 

Survey  duration: 

11.0  hours 

Electromagnetic  interpretation  noise  threshold: 

+10  emu 

Number  of  interpreted  metallic  items: 

416 

Total  number  of  targets  identified: 

828  (combined  data  set) 
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TABLE  2 

Artillery  and  Mortar  Range 


Total  area  surveyed  (Magnetics): 

4.68  hectares 

(Electromagnetics): 

4.53  hectares 

Magnetic  sensor  elevation: 

0.65  m 

Magnetic  sample  interval  along  traverses: 

0.1  m 

Magnetic  sample  interval  across  traverses: 

0.5  m 

Number  of  magnetic  data  measurements: 

827,820 

Survey  duration: 

14  hours 

Magnetic  interpretation  noise  threshold: 

0.2  nT 

Number  of  interpreted  ferrous  items: 

1,186 

Electromagnetic  sensor  elevation: 

0.05  m 

Electromagnetic  sample  interval  along  traverses: 

0.05  m 

Electromagnetic  traverse  width: 

0.5  m 

Number  of  electromagnetic  data  measurements: 

1,655,640 

Survey  duration: 

15  hours 

Electromagnetic  interpretation  noise  threshold: 

+10  emu 

Number  of  interpreted  metallic  items: 

611 

Total  number  of  targets  identified: 

1,280  (combined  data 

set) 
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TABLE  3 

Grenades  and  Sub-munitions  Range 


Total  area  surveyed  (Magnetics): 

(Electromagnetics) : 

Magnetic  sensor  elevation: 

Magnetic  sample  interval  along  traverses: 
Magnetic  sample  interval  across  traverses: 
Number  of  magnetic  data  measurements; 

Survey  duration: 

Magnetic  interpretation  noise  threshold: 

Number  of  interpreted  ferrous  items: 

Electromagnetic  sensor  elevation: 
Electromagnetic  sample  interval  along  traverses: 
Electromagnetic  traverse  width: 

Number  of  electromagnetic  data  measurements: 
Survey  duration: 

Electromagnetic  interpretation  noise  threshold: 
Number  of  interpreted  metallic  items: 


4.16  hectares 
4.08  hectares 

0.65  m 
0.1  m 
0.5  m 
934,690 
13.5  hours 
0.2  nT 
853 

0.05  m 
0.05  m 
0.5  m 
1,869,380 
14  hours 
+10  emu 
694 


Total  number  of  targets  identified: 
set) 


1,071  (combined 


data 


10.2.5  Digital  Data 

10.2.5.1  Raw  Data 

The  raw  data  were  provided  in  XYZ  format,  local  coordinates.  Coordinate  translation  was 
used  to  provide  UIM  coordinates.  The  parameters  used  in  this  translation  are  listed  in 
the  file  ‘readme.txt’. 

•  The  magnetic  raw  data  from  the  Aerial  Gunnery  Range  is  contained  in  file  IMG.XYZ 

•  The  magnetic  raw  data  from  the  Artillery  and  Mortar  Range  is  contained  in  file 
2MG.XYZ 

•  The  magnetic  raw  data  from  the  Grenades  and  Sub-munitions  Range  is  contained  in 
file  3MG.XYZ 

•  The  EM  raw  data  from  the  Aerial  Gunnery  Range  is  contained  in  file  1EM.XYZ 

•  The  EM  raw  data  from  the  Artillery  and  Mortar  Range  is  contained  in  file  2EM.XYZ 

•  The  EM  raw  data  from  the  Grenades  and  Sub-munitions  Range  is  contained  in  file 

3EM.XYZ 

% 

10.2.5.2  Processed  Data 

The  combined  magnetic  and  electromagnetic  interpretation  database  (GTLCB.XLS)  is 
our  primary  processed  data  set.  The  databases  for  each  individual  interpretation  has  been 
provided  to  facilitate  a  breakdown  analysis  from  which  the  value  of  the  components  and 
the  combination  may  be  assessed. 

•  The  complete  magnetic  target  database  is  contained  in  file  GTLMG.XLS 

•  The  complete  EM  target  database  is  contained  in  file  GTLEM.XLS 

•  The  complete  combined  target  database  is  contained  in  file  GTLCB.XLS 

10.2.6  Conclusions 

•  Four  hectares  at  each  of  the  “Aerial  Gunnery  Range”,  “Artillery  and  Mortar  Range” 
and  “Grenades  and  Sub-munitions  Range”  sites  were  mapped  with  GTL’s  proprietary 
TM-4,  quad  sensor  magnetometer  system  and  proprietary  TM-4e  electromagnetic 
system. 

•  The  magnetic  survey  was  conducted  with  a  single,  two  person  operated  instrument  in  a 
total  survey  time  of  38  hours  with  2  hours  down-time. 

•  The  electromagnetic  survey  was  conducted  by  three,  single-person  operated  TM-4e 
instruments  in  a  total  crew  time  of  120  hours.  No  down-time  was  experienced. 

•  The  magnetic  survey,  optimised  for  targets  occurring  deeper  than  0.3  m,  detected  a 
total  of  2,919  magnetic  targets  on  this  data  alone. 

•  The  EM  survey  detected  a  total  of  1,721  metallic  targets. 

•  Integrating  the  magnetic  and  EM  data  permitted  discrimination  against  false  negative 
magnetic  targets  and  resulted  in  a  total  of  3,179  potential  targets  being  identified. 

Integration  of  the  magnetic  and  electromagnetic  data  optimised  the  deep  search  capability 
of  the  TM-4  with  the  ability  of  the  TM-4e  to  detect  non-ferrous  metals  and  to 
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discriminate  against  interference  from  mineralised  soils. 
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GRI  (Combined)  -  Combined  Statistics:  Scenarios  1,2,3 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Pd” 

Ordnance 

208 

194 

0.93 

0.276 

Nonordnance 

166 

147 

0.89 

1 

Total 

374 

341 

Number  False  Alarms 

2955 

False  Alarm  Rate  (#/Hectare) 

240.53 

False  Alarm  Ratio  (#/Ord.) 

15.23 

Probability  False  Alarms 

0.302 

Locaiization  Statistics  (TMA  Closest)  (In  meters) 


Mean 

Std  Deviation  | 

dx  •  eastina  error 

-0.04 

0.3 

dv  -  northing  error 

0.08 

0.35 

Radial  error 

0.39 

0.27 

iDepth  (z) _ 

dz  -  averaged  depth  error 

0.23 

0.34  1 

Idzl®-  absolute  depth  error 

0.41 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Po** 

#  Correct 

Pc” 

Abilitv  to  Type 

Ordnance 

208 

194 

0.93 

182 

0.94 

Nonordnance® 

167 

147 

0.88 

16 

0.11 

14 

14 

1.00 

7 

0.50 

Medium 

39 

37 

0.95 

7 

0.19 

Smail 

155 

143 

0.92 

60 

0.42 

Abilitv  to  Classify 

Bomb 

21 

21 

1.00 

10 

0.48 

Projectile 

42 

39 

0.93 

11 

0.28 

Mortar 

26 

22 

0.85 

10 

0.45 

Submunition 

97 

92 

0.95 

0 

0.00 

Rocket 

22 

20 

0.91 

10 

0.50 

Notes:  *  Target  Matching  Algorithm 

"  Probability  of  detection 
’’Square  root  of  the  mean  square  depth  error 
‘‘Probability  of  correct  characterization 
^MA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 
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JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator:  GRI-Combined 

Small  Medium  Lai 
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■  Target  Detected 
□  Target  Not  Detected 


GRI  (EM)  -  Combined  Statistics:  Scenarios  1,2,3 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Ordnance 

208 

180 

0.87  I  0.16 

Nonordnance 

166 

135 

0.81  J 

Total 

374 

315 

Number  False  Alarms 

1522 

False  Alarm  Rate  (#/Hectare) 

123.89 

False  Alarm  Ratio  (#/Ord.) 

8.46 

Probability  False  Alarms 

0.156 

Localization  Statistics  (TMA  Closest)  (In  meters) 


Mean 

Std  Deviation 

Position  (x,v) 

dx  -  eastinq  error 

-0.04 

0.32 

dv  -  northinq  error 

0.1 

0.33 

Radial  error 

0.39 

0.28 

Depth  (z) 

dz  -  averaged  depth  error 

0.32 

0.36 

Idzl®-  absolute  depth  error 

0.48 

Characterization  Statistics  (TMA  Closest) 


1  #  Baseline  |  #  Detected  |  Pd°  |#  Correct  |  Pc"  I 

1 1  lllli'il  ti'l ' 

Ordnance 

208 

180 

0.87 

178 

0.99 

Nonordnance® 

167 

135 

0.81 

7 

0.05 

lAbilitv  to  Size  _ 1 

Larqe 

14 

6 

0.43 

0 

0.00 

Medium 

39 

34 

0.87 

0 

0.00 

Small 

155 

140 

0.90 

0 

0.00 

lAbititv  to  Classify  1 

Bomb 

21 

13 

0.62 

0 

0.00 

Projectile 

42 

37 

0.88 

0 

0.00 

Mortar 

26 

22 

0.85 

0 

0.00 

Submunition 

97 

89 

0.92 

0 

0.00 

Rocket 

22 

19 

0.86 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

'*  Probability  of  detection 
‘Square  root  of  the  mean  square  depth  error 
"  Probability  of  correct  characterization 
‘TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 
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Depth  (meters) 


JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator:  GRI-EM 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 


GRI  (Mag)  -  Combined  Statistics:  Scenarios  1,2,3 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Pd“ 

Ordnance 

208 

146 

0.70 

0.257  1 

Nonordnance 

166 

140 

0.84 

Total 

374 

286 

Number  False  Alarms 

2748 

False  Alarm  Rate  (#/Hectare) 

223.68 

False  Alarm  Ratio  (#/Ord.) 

18.82 

Probability  False  Alarms 

0.281 

Locaii2atlon  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

. II  11  1— ^^1^— ^ 

dx  -  eastinq  error 

-0.07 

0.37 

dy  -  northing  error 

0.08 

0.46 

Radial  error 

0.48 

0.35 

iDepth  (z) _ _ _ 

dz  -  averaged  depth  error 

0.14 

0.44  1 

Idzl®-  absolute  depth  error 

0.46 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Po" 

#  Correct 

Pc- 

Ordnance 

208 

146 

0.70 

128 

0.88 

Nonordnance® 

167 

140 

0.84 

16 

0.11 

Large 

14 

14 

1.00 

7 

0.50 

Medium 

39 

36 

0.92 

8 

0.22 

Small 

155 

96 

0.62 

87 

0.91 

kbilitv  to  Classify  - 1 

Bomb 

21 

21 

1.00 

10 

0.48 

Projectile 

42 

39 

0.93 

14 

0.36 

Mortar 

26 

24 

0.92 

20 

0.83 

Submunition 

97 

45 

0.46 

0 

0.00 

Rocket 

22 

17 

0.77 

12 

0.71 

Notes:  *  Target  Matching  Algorithm 

Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  •  not  applicable 


B.  10-14 


JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator:  GRI-Magnetometer 

Small  Medium  Large 


Principal  Diameter  (millimeters) 


■  Target  Detected 
□  Target  Not  Detected 
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11.0  LOCKHEED  MARTIN  ADVANCED  ENVIRONMENTAL 
SYSTEMS^^ 

The  Caterpillar  320L  Low  Ground  Pressure  excavator  with  remote  operator  control  station  was 
demonstrated  as  a  remediation  platform  at  the  JPG  16-hectare  demonstration  site  from  November 
20  to  22, 1996.  Lockheed  Martin  was  a  first  time  participant  in  the  UXO  ATD  program. 

11.1  TECHNOLOGY  DESCRIPTION 

11.1.1  System  and  Transport  Mode 

The  LMAES  system  included  a  Caterpillar  320L  Low  Ground  Pressure  (LGP)  excavator, 
and  one  remote  operator  control  station  (OCS)  located  in  a  panel  van.  The  320  was 
outfitted  with  a  1.125  cubic  yard  bucket  and  hydraulic  thumb  for  excavation  and  removal 
of  ordnance.  The  standard  320  had  been  modified  with  the  LMAES  Reconfigurable 
Remote  Control  System  (R2CS)*  electronics  package  to  allow  tele-operation  of  the 
vehicle.  The  OCS  consisted  of  a  chair,  operator  controls,  audio,  video  and  computer 
displays,  and  electronics  which  enabled  the  operator  to  remotely  control  the  excavator. 
The  OCS  used  spread-spectrum  radios  for  command  and  control  of  the  excavator.  A 
separate  RF ’'channel  provides  the  audio  and  video,  while  a  dedicated  narrowband  RF 
signal  was  used  for  range  safety.  For  precision  navigation,  a  DGPS  system  was  installed 


Lockheed  Martin  Advanced  Environmental  Systems,  Jefferson  Proving  Ground 
Advanced  Technology  Demonstrations  Phase  IH,  Demonstration  Summary  Report, 
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on  the  320.  A  dedicated  computer  display  at  the  OCS  provides  the  operator  with  real¬ 
time  feedback  on  vehicle  position.  A  custom  antenna  mount  was  developed  to  align  the 
DGPS  antenna  with  the  bucket,  so  that  the  bucket  could  be  positioned  as  accurately  as 
possible  over  the  target. 

11.1.2  Recommended  Applications  and  Technology  Limitations 

LMAES  developed  the  remote  excavator  for  remediation  and  other  hazardous  work.  The 
remote  excavator  has  direct  application  to  the  numerous  DoD  cleanup  programs  at  current 
or  former  military  installations.  These  cleanup  programs  include  the  unearthing,  handling 
and  removal  of  stored  and  buried  ordnance.  There  are  no  known  limitations  to  the  use  of 
this  technology  for  the  given  application. 

11.1.3  Logistics  Required 

Very  little  in  the  way  of  logistics  is  required  to  support  the  LMAES  remote  excavator.  As 
a  daily  procedure^,  the  vehicle  should  be  fueled  and  daily  maintenance  performed  prior  to 
operations.  The  vehicle  can  carry  enough  fuel  to  support  a  full  8  hour  work  day.  The 
OCS,  as  packaged,  is  powered  by  a  5000  watt  generator  whose  fuel  tank  is  also  sufficient 
to  support  an  8  hour  work  day. 

11.1.4  Data  Acquisition 

Performance  data  for  these  demonstrations  was  recorded  manually  by  the  test  director 
from  the  feedback  provided  to  the  operator  on  the  various  displays.  The  OCS  VCRs  were 
also  used  to  videotape  operations. 

11.1.5  Data  Processing  and  Interpretation 

All  data  processing,  specifically  for  the  DGPS  system,  was  performed  manually. 

11.1.6  Quality  Assurance 

To  provide  Quality  Assurance,  LMAES  provided  a  certified  320L  operator  as  well  as  a 
highly  skilled  technician  for  on-site  operations.  A  notebook  of  all  operations  was 
maintained  by  the  test  director  to  document  the  events  which  transpired  while  on-site  and 
a  library  of  videotapes  has  been  made  from  the  vehicle  and  site  video  VCRs. 

11.2  DEMONSTRATION  RESULTS 
11.2.1  Assumptions 

LMAES  assumed  that  all  buried  ordnance  at  the  JPG  site  were  inert.  Therefore,  formal 
procedures  or  protocols  for  handling  hazardous  materials  were  not  instituted. 
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11.2.2  Site-Specific  Procedures 

The  basic  remediation  procedure  developed  by  LMAES,  first  had  the  operator  navigate 
the  excavator  towards  the  target  as  displayed  on  the  DGPS  display.  After  the  excavator 
got  close  to  the  target,  the  display  was  zoomed  in,  and  the  bucket  oriented  so  that  it  was 
directly  over  the  target.  Once  the  bucket  was  positioned,  the  bucket  teeth  were  used  to 
scratch  the  ground.  The  excavator  was  then  repositioned  for  digging  depending  on  the 
expected  depth  of  the  target.  The  digging  sequence  was  a  series  of  scratching  with  the 
teeth  to  break  the  dirt  loose,  and  then  removing  the  dirt  with  the  bucket  until  the  target 
was  visible.  Once  visible,  the  bucket  and  thumb  were  used  to  extract  the  ordnance  from 
the  hole.  The  ordnance  was  separated  from  any  dirt,  set  on  the  ground,  and  the  hole  filled 
back  in. 

11.2.3  Problems  Encountered 

LMAES  encountered  three  problems  during  the  course  of  the  demo.  First,  was  confusion 
over  the  map  coordinates  supplied  as  to  whether  they  were  in  WGS-84  or  State  Plane 
projections.  Secondly,  the  operator  control  van  had  to  be  repositioned  on  day  2  to 
maintain  line-of-sight  with  the  excavator.  LOS  was  necessary  for  the  radio  configuration 
used.  Thirdly,  an  emergency  stop  occurred  on  the  excavator  during  day  3  as  a  result  of  a 
tripped  circuit  breaker.  The  cause  for  the  breaker  tripping  was  not  identified. 

11.2.4  Discussion  of  Resuits 

11.2.5  Remediation  History 

During  the  first  day  of  the  demonstration,  Wednesday  November  20th,  the  command  and 
control  station  was  set  up  using  the  temporary  monument  located  behind  the  trailer  on  the 
east  perimeter  road  about  one-half  mile  from  the  grid.  At  this  location  we  were  limited  to 
remediating  targets  on  the  east  side  of  scenario  4,  because  line  of  sight  communication 
with  the  excavator  was  lost  due  to  elevation  drop  and  tree  cover  on  the  grid.  On 
Thursday  morning,  November  21st,  the  command  and  control  station  was  relocated  onto 
the  grid  and  setup  using  Monument- 1.  It  remained  there  for  the  rest  of  the  demonstration. 

20-Nov,  Num.  1,  Target  1219, 4.2in  mortar,  0.38m  (1.25ft)  deep. 

Forty-nine  minutes  into  the  excavation  a  metal  detector  was  brought  in  to  detemune  if 
the  mortar  was  still  in  the  hole.  It  was  located  in  the  dirt  removed  from  the  hole.  The 
operator  then  sifted  through  the  dirt  using  the  teeth  of  the  bucket.  The  metal  detector  was 
brought  in  a  second  time,  the  mortar  was  located  and  marked  with  a  stake.  Sixty-nine 
minutes  into  the  excavation  the  mortar  was  located  by  the  operator  using  the  cameras. 
The  operator  then  successfully  picked  up  the  mortar  using  the  bucket  and  the  thumb. 

20- Nov,  Num.  2,  Target  1258,  8ia  projectile,  1.37m  (4.49ft)  deep. 

Thirty-nine  minutes  into  the  excavation  the  projectile  was  located  by  the  operator  using 
the  cameras.  The  operator  then  successfully  picked  up  the  projectile  using  the  bucket  and 
the  thumb. 

21- Nov,  Num.  3,  Target  1215,  175mm  projectile,  1.31m  (4.30ft)  deep. 


B.11-3 


Thirty-eight  minutes  into  the  excavation  a  metal  detector  was  brought  in  to  determine  if 
the  projectile  was  still  in  the  hole.  It  was  located  in  the  hole.  Eighty  minutes  into  the 
excavation  the  projectile  was  located  by  the  operator  using  the  cameras.  Three  minutes 
later  it  was  removed  from  the  hole. 

21-Nov,  Num.  4,  Target  1253,  152mm  projectile,  0.58m  (1.90ft)  deep. 

Fifty-five  minutes  into  the  excavation  a  visual  check  was  made  at  the  hole  to  locate  the 
projectile.  The  projectile  was  found  on  the  surface  behind  the  dirt  pile,  out  of  the 
operator  s  view. 

21-Nov,  Num.  5,  Target  1263,  500-lb  bomb,  0.98m  (3.22ft)  deep. 

Three  minutes  into  the  excavation  the  operator  had  the  bomb  in  the  bucket.  Total  number 
of  scoops  during  the  excavation  was  three. 

21-Nov,  Num.  6,  Target  1272,  250-lb  bomb,  1.82m  (5.97ft)  deep. 

Twenty-four  minutes  into  the  excavation  the  bomb  was  located  by  the  operator  using  the 
cameras.  The  operator  dropped  the  bomb  on  the  first  attempt  to  remove  it  from  the  hole. 
The  operator  successfully  removed  the  bomb  two  minutes  after  locating  it  in  the  hole. 

21- Nov,  Num.  7,  Target  1577,  5in  rocket,  0.42m  (1.38ft)  deep. 

Eleven  minutes  into  the  excavation  the  operator  uncovered  a  suspect  object.  Three 
minutes  later  EOD  personnel  determined  from  visual  inspection  at  the  hole  that  the  rocket 
casing  had  been  sheared  into  two  pieces.  Eighteen  minutes  into  the  excavation  the 
operator  removed  the  largest  piece  of  the  rocket  casing. 

22- Nov,  Num.  8,  Target  1269,  500-lb  MK-82,  1.44m  (4.72ft)  deep. 

Seventeen  minutes  into  the  excavation  the  bomb  was  located  by  the  operator  using  the 
cameras.  Five  minutes  later  the  operator  had  the  bomb  on  the  ground  out  of  the  hole. 

22-Nov,  Num.  9,  Target  1229,  500-Ib  bomb,  1.07m  (3.51ft)  deep. 

Twenty  minutes  into  the  excavation  the  operator  was  informed  by  EOD  personnel  that  he 
had  been  driving  the  bomb  into  the  near  wall  of  the  hole  (toward  the  excavator)  since  the 
fourth  bucket  of  dirt  was  removed.  Five  minutes  later  the  operator  removed  the  bomb 
from  the  hole. 

22-Nov,  Num,  10,  Target  1235,  750-lb  bomb,  1.17m  (3.84ft)  deep. 

Fourteen  minutes  into  the  excavation  the  bomb  was  located  by  the  operator  using  the 
cameras.  Four  minutes  later  the  operator  had  the  bomb  on  the  ground  out  of  the  hole. 

22-Nov,  Num.  11,  Target  231,  500-lb  MK-82, 3.1  Im  (10.20ft)  deep. 

Twenty  minutes  into  the  excavation  the  operator  began  widening  the  hole  to  the  right  and 
taking  another  foot  off  the  far  wall.  Fifty  -eight  minutes  into  the  excavation  the  bomb 
was  located  by  the  operator  using  the  cameras.  Three  minutes  later  the  operator  had  the 
bomb  on  the  ground  out  of  the  hole.  Total  excavation  time,  after  subtracting  sixteen 
minutes  for  an  emergency  stop  delay,  was  forty-five  minutes. 
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11.2.6  Navigation  History 

The  real-time  DGPS  system  was  used  to  navigate  to  eleven  marked  UXO  targets  and  two 
unmarked  UXO  targets.  Latitude  and  longitude  of  the  surveyed  targets  were  compared  to 
the  latitude  and  longitude  of  the  DGPS  system  once  positioned  over  the  target.  The 
differences  are  shown  in  Table  1.  The  accuracy  of  the  DGPS  system  is  two  to  five  cm. 
The  remainder  of  the  error  is  a  function  of  how  accurately  the  operator  can  place  the 
DGPS  antenna,  and  therefore  the  bucket,  over  the  target.  Navigation  to  the  marked  targets 
was  successful  eleven  out  of  twelve  times.  Navigation  to  the  first  target  failed  because 
the  target  entered  into  the  DGPS  map  was  in  error  by  one  second  in  latitude.  Navigation 
to  the  unmarked  targets  was  successful  both  times.  Error  for  the  first  unmarked  target 
was  less  than  one  inch  and  for  the  second  target  was  one  foot. 

The  real-time  DGPS  system  was  also  used  to  determine  the  depth  of  the  deeper  holes 
during  excavation. 

In  hole  Number  2  it  was  used  to  check  the  depth  of  the  hole  after  excavation  was 
complete.  EOD  personnel  estimated  the  hole  to  be  between  five  and  six  feet  deep.  The 
DGPS  systems  elevation  delta  between  the  surface  and  the  bottom  of  the  hole  was  5.5 
feet  (  904.8v-  899.3  ).  For  hole  Number  11  the  DGPS  elevation  was  used  during 
excavation  as  an  additional  feedback  to  the  operator  s  camera  views.  The  operator 
estimated,  knowing  the  length  of  the  stick  and  the  size  of  the  bucket,  the  hole  was  thirteen 
feet  deep.  The  elevation  delta  of  the  DGPS  system  was  13.2  feet  ( 915.7  -  902.5  ). 

11.2.7  DGPS  Performance 

DGPS  system  performance  was  excellent.  During  the  four  days  of  testing  and 
demonstrating  the  system  only  lost  satellite  lock  once.  This  could  have  been  due  to 
incomplete  initialization  (  operator  error  )  and/or  a  brief  period  of  bad  satellite  geometry. 
During  excavation  the  antenna  s  orientation  changed  an  estimated  ten  to  thirty  degrees 
while  maintaining  satellite  lock. 

The  Geolink  DGPS  software  used  for  vehicle  tracking  and  navigation  worked  well.  The 
cursor  representing  the  vehicle  was  easily  visible.  A  Geolink  software  utility  was  used  to 
create  graphic  scaleable  symbols  for  targets  from  lat./long.  coordinates.  The  dotted  trail 
generated  during  travel  was  used  as  a  heading  indicator  to  easily  navigate  toward  a  target. 

11.2.8  Excavator  Performance 

From  a  reliability  standpoint  the  tele-operated  excavator  had  already  proven  itself  during 
the  past  five  months  of  contracted  work  at  Aberdeen  Proving  Ground  (APG)  in  Maryland. 
It  continued  to  perform  well  during  the  demonstration  at  JPG.  During  the  three  days  of 
operation  two  minor  incidences  occurred.  A  hydraulic  leak  was  discovered  at  the  end  of 
the  day  on  Thursday.  The  hose  was  replaced  first  thing  Friday  morning.  Down  time  for 
the  excavator  was  forty-five  minutes.  Friday  afternoon  an  Emergency  Stop  occurred 
shutting  down  the  excavator.  During  diagnosis  of  the  problem  the  CPU  circuit  breaker 
was  reset.  This  corrected  the  problem  and  lead  us  to  believe  we  had  a  faulty  circuit 
breaker.  Down  time  for  the  excavator  was  sixteen  minutes. 
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From  a  control  standpoint  the  operator  has  nearly  the  same  capability  remotely  as  he  does 
in  the  cab.  During  tele-operated  control  the  boom,  stick,  bucket  and  swing  have  the  same 
feel  as  in  the  cab.  The  foot  pedals  have  a  slightly  different  feel  traveling  at  slow  speeds. 
The  camera  views  the  operator  has  give  him  more  capability  than  a  man  in  the  cab  has 
while  excavating.  He  uses  the  pan,  tilt  and  zoom  for  close-up  views  inside  the  bucket  and 
inside  the  hole.  The  camera  views  the  operator  has  while  traveling  limit  his  side  to  side 
view.  This  is  a  limitation  the  operator  has  easily  adapted  to  and  it  has  not  posed  any 
problems  during  tele-operated  travel. 

11.2.9  Excavation  Performance 

A  tele-operated  excavation  performance  study  was  done  to  quantify  how  much  dirt  can 
be  moved  over  what  period  of  time.  The  data  used  was  the  time  tagged  video  tape 
recorded  while  UXO  target  Number  1 1  was  being  remediated.  The  target  was  buried  10.2 
feet  deep.  Data  was  collected  over  the  first  thirteen  minutes  of  the  excavation.  Elapsed 
time  was  recorded  for  fifteen  individual  scoops.  The  duration  of  a  scoop  is  from  when 
the  teeth  go  below  the  surface,  a  bucket  full  of  dirt  is  removed  and  dumped  on  the  ground, 
the  bucket  is  positioned  for  the  next  scoop  and  the  teeth  go  below  the  surface  again.  The 
minimum  time  for  a  scoop  was  thirty-three  seconds  while  the  maximum  was  seventy-one 
seconds.  The^average  scoop  was  calculated  at  52.7  seconds/scoop  or  1.14  scoops/minute. 

The  amount  of  dirt  in  the  bucket  for  each  scoop  was  estimated.  Measurement  criteria  was 
percent  full  recorded  in  ten  percent  increments.  All  of  the  fifteen  buckets  were  overfilled. 
The  percent  overfill  ranged  from  ten  to  one  hundred  with  an  average  overfill  of  forty 
percent.  Using  a  1.125  cubic  yard  bucket,  a  cubic  yards  per  hour  rate  can  be  calculated 
for  the  amount  of  dirt  being  removed  from  the  hole.  Two  rates  were  calculated  to  present 
a  performance  range.  Using  zero  percent  overfill  the  rate  was  77  cubic  yards/hour.  Using 
the  estimated  forty  percent  overfill  the  rate  was  108  cubic  yards/hour. 

11.2.10  Digital  Data 

Appendix  A  contains  the  raw  data  recorded  by  LMAES  for  each  scenario.  This  appendix 
has  additional  information  not  required  by  the  official  data  reporting  sheet.  Appendix  B 
contains  the  formal  report  data  in  the  specified  format  for  the  JPG  III  database. 

11.2.11  Conclusions 

We  have  demonstrated  at  JPG,  and  the  previous  five  months  at  APG,  that  tele-operated 
excavation  is  effective.  The  first  step  has  been  taken  in  applying  this  technology  for  the 
remediation  of  buried  UXO  s.  By  developing  our  recommended  improvements  we  will 
be  one  step  closer  to  fielding  a  production  level  system  capable  of  undertaking  large 
remediation  projects. 

11.2.12  References 

'  “R2CS  -  Reconfigurable  Remote  Control  System”,  Laura  Dussinger  and  Scott 
Williams,  to  be  published  by  the  American  Nuclear  Society,  in  the  Proceedings  of  the 
7th  Topical  Meeting  on  Robotics  and  Remote  Systems,  May  1997. 
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12.0  NAEVA  GEOPHYSICS  INC.  ^ 

NAEVA  Geophysics  demonstrated  from  September  11  through  15,  1996  at  the  16-hectare 
area  at  JPG.  NAEVA  Geophysics  was  a  first  time  participant  in  the  UXO  ATD  program. 

12.1  TECHNOLOGY  DESCRIPTION 

12.1.1  Sensor  System  and  Transport  Mode 

NAEVA  Geophysics  surveyed  a  10  acre  Aerial  Gunnery  Range  and  a  10  acre  Artillery 
and  Mortar  Range  using  man-portable  Scintrex  Smartmag  SM-4  total  field 
magnetometers  and  Geonics  EM-61  metal  detection  instruments. 


The  Scintrex  Smartmag  SM-4  system  is  based  upon  a  very  sensitive  self-oscillating  split- 
beam  cesium  vapor  magnetometer.  It  measures  the  total  magnetic  field  with  a  sensitivity 
of  ±  0.01  nT  (range  15,000  to  100,000  nT)  at  sample  rates  from  1  to  10  samples  per 
second.  The  SM-4  system  includes  a  cesium  sensor,  associated  electronics,  carrying 
harness,  ENVI  control  console,  ENVIMAP  operating  software,  and  rechargeable 
batteries.  The  system  is  man-portable. 

^  NAEVA  GEOPHYSICS  INC.,  Phase  III  UXO  Detection,  Advanced  Technology 
Demonstration,  Jefferson  Proving  Grounds,  Madison  Indiana  -  Survey  Data 
Analysis  Report 
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The  Geonics  EM-61  is  a  time-domain  electromagnetic  instrument  designed  to  detect 
shallow  metallic  objects  with  good  spatial  resolution.  The  system  consists  of  two  air- 
cored  coils,  each  one  meter  in  diameter,  batteries  and  processing  electronics,  and  a  digital 
data  recorder.  Secondary  voltages  induced  in  both  coils  are  measured  in  millivolts.  The 
coils  are  arranged  so  that  the  larger  coil  (EM  source  and  receiver)  lies  40  cm  below  a 
second  receiver  coil.  Two  modes  of  operation  are  available,  1)  trailer  mode,  in  which  two 
wheels  support  the  coils,  and  2)  harness  mode,  in  which  the  operator  carries  the  coils  on  a 
belt  and  shoulder  harness.  Three  instruments  may  be  "ganged"  and  pulse  synchronized  in 
order  to  cover  open  ground  rapidly  with  one  meter  line  spacings. 

12.1.2  Recommended  Applications  and  Technology  Limitations 

The  current  system  is  well  suited  for  ordnance  detection  on  sites  where  unexploded 
surface  ordnance  have  been  cleared.  The  EM-61  metal  detectors  are  designed  for 
resolving  small  shallow  features  or  non-ferrous  metallic  ordnance,  while  the 
magnetometer  is  preferred  for  detection  of  large,  deeper  ferromagnetic  ordnance  (beyond 
the  range  of  reliable  EM-61  metal  detection).  This  integrated  approach  achieved  better 
results  than  either  method  standing  alone.  The  EM-6 1  is  more  affected  by  topography 
than  the  magnetometer,  because  it  is  most  commonly  operated  with  wheel-mounted  coils. 
For  both  systems,  surveying  in  areas  of  thick  vegetation  requires  prior  line  clearing.  The 
only  geologic  limitation  imposed  on  the  system  is  naturally  occurring  high  variation  in 
magnetic  background.  The  EM-61  is  virtually  unaffected  by  soil/geologic  conditions. 
Heavy  rain  may  restrict  operation  of  both  instruments. 

12.1.3  Logistics  Requirements 

NAEVA  Geophysics'  geophysical  ordnance  detection  system  can  be  employed  with 
minimal  logistical  requirements.  The  instruments  can  be  shipped  through  standard 
commercial  services  world-wide.  The  crew  size  is  dependant  on  the  area  of  investigation 
and  can  be  as  small  as  two  persons. 

12.1.4  Data  Acquisition 

The  Aerial  Gunnery  Range  (1)  and  the  Artillery  and  Mortar  Range  (2)  subareas  were 
swept  using  three  Geonics  EM-61  metal  detectors  and  a  Scintrex  Smartmag  cesium  vapor 
magnetometer. 

The  ganged  EM-61  instruments  swept  a  10-foot  width  with  a  sensor  separation  of 
approximately  one  meter.  The  Smartmag  collected  data  at  5-foot  line  separation.  The 
magnetometer  operated  in  continuous  mode.  The  EM-61  operated  in  wheel  odometer 
mode,  collecting  data  every  0.63  feet  along  lines.  Data  were  directly  recorded  for  digital 
analysis,  and  gridded  to  a  1-foot  interval.  A  separate  base  station  magnetometer  acquired 
data  for  magnetic  diurnal  drift  corrections. 
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Hand-held  FM  radios  were  used  for  communications  by  the  five  person  field  crew  and 
PRC.  The  survey  lines  were  controlled  by  chain  on  10-foot  spaced  north-south  survey 
grid  lines  with  fiducials  every  20  feet,  in  order  to  assure  better  than  ±  one  foot  accuracy. 

12.1.5  Data  Processing  and  Interpretation 

The  magnetic  data  were  processed  and  analyzed  using  the  MAGFIT  method.  MAGFIT 
has  been  devised  and  implemented  in  computer  C++  code  for  UNIX  and  DOS  operating 
systems  by  Dr.  G.  Hunter  Ware  and  Hunter  A.  Ware.  This  unique  and  proprietary 
computer  program  scans  the  theoretical  anomalies  of  a  very  large  number  of  magnetic 
dipole  models  (all  locations,  depths,  orientations,  and  dipole  moments  of  interest)  over 
the  field  data,  and  identifies  the  best  models  using  a  "best  least  squares  fit"  criteria. 
MAGFIT  also  yields  model  fit  contours  (in  1%)  around  the  best  fit  location,  in  plan  or 
cross-section.  This  technique  has  only  recently  become  feasible,  due  to  increases  in  the 
speed  and  memory  capacity  of  small  computers.  MAGFIT  was  described  in  greater  detail 
by  G.H.  Ware  and  H.A.  Ware  in  a  recent  paper  published  in  the  proceedings  of  SAGEEP 
1996. 

MAGFIT  is  superior  to  commercially  available  software  for  total  field  magnetic  data 
analysis  whidh  use  Euler  deconvolution,  "analytic  function"  analysis,  or  interactive 
modeling.  These  approaches  require  the  numerical  calculation  of  field  derivatives 
(gradients),  which  is  often  inaccurate  for  sparsely  gridded  data.  It  is  preferable  to  select  a 
large  number  of  simple  (but  appropriate)  models  such  as  dipoles,  and  scan  their 
anomalies  over  the  actual  field  data,  seeking  best  least-squares  fits.  MAGFIT  avoids  the 
numerical  approximations  of  the  analytic  methods. 

EM-61  profiles  were  interpreted  using  Geonics  DAT-61  software.  Depth  and  character 
estimates  by  MAGFIT  and  DAT-61  were  integrated  in  order  to  achieve  more  accurate 
final  predictions  of  target  location  and  depth. 

12.1.6  Quality  Assurance 

Data  quality  was  ensured  by:  1)  experienced  field  personnel;  2)  use  of  a  base  station 
magnetometer  to  eliminate  the  effects  of  diurnal  drift;  3)  instrument  calibration,  as 
required;  4)  repetition  of  selected  north-south  survey  lines  and  east-west  tie  lines  for  data 
comparison;  and  5)  daily  review  of  geophysical  data  profiles  and  contours. 

12.2  Demonstration  Results 

12.2.1  Assumptions 

It  was  assumed  that  the  100  by  100  foot  control  grid  previously  emplaced  is  properly 
located  and  accurate.  It  is  also  assumed  that  target  sizes  and  maximum  depth  of  burial  for 
each  scenario  has  been  accurately  described. 
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12.2.2  Site  Specific  Procedures 


Survey  lines  were  established  over  designated  grid  blocks  by  placing  marked  ropes  east- 
west,  spaced  20  feet  apart.  Wooden  stakes  were  used  to  secure  the  ropes,  although  an 
alternative  method  would  be  used  if  live  ordnance  is  suspect.  Instruments  were  run 
simultaneously  in  a  systematic  fashion  across  the  grid  blocks.  The  EM-bl's  were  run  as  a 
three  ganged  unit  in  open  areas  and  individually  in  areas  of  scattered  trees. 

12.2.3  Problems  Encountered 

Survey  procedures  required  use  of  existing  grid  nodes  (marked  by  PVC  tubes  and  stakes) 
for  grid  control.  Some  minor  inaccuracies  were  noted  when  directly  measuring  between 
nodes.  Although  minimal,  these  discrepancies  may  introduce  a  source  of  error  in 
determining  location. 

A  moderate  amount  of  instrument  noise  was  recognized  in  both  EM-61  and  magnetics 
data,  necessitating  additional  post-collection  processing. 

12.2.4  Discyssion  of  Results 

¥ 

With  our  integrated  magnetic  and  electromagnetic  surveys,  we  have  selected  141 
ordnance  targets  in  scenario  1  and  143  in  scenario  2. 

Field  data  were  measured  along  grid  lines  with  fiducials  placed  every  20  feet.  The 
precision  of  measurement  locations  is  probably  ±  1  to  2  feet,  due  to  uncertainties  in 
sensor  velocity  and  fiducial  location.  These  location  errors  and  terrain  noise  are  probably 
the  limiting  factors  in  accurate  target  location. 

12.2.5  Conclusions 

We  believe  the  technologies  that  were  demonstrated  are  appropriate  and  complimentary. 
It  is  possible  that  we  have  interpreted  some  small  EM  anomalies  as  ordnance,  that  are  in 
fact  nonordnance  or  false  positives.  This  is  because  we  do  not  have  actual  ordnance  with 
which  to  calibrate  our  instrument  response.  We  would  appreciate  the  opportunity  to 
calibrate  our  measurements  and  target  selections  against  the  site  truth  table  at  some  future 
date. 


12.2.6  References 

Magnetic  Interpretation  by  Scanning  Multiple  Models  for  Best  Least  Squares  Fit,  Ware, 
G.H.,  and  Ware,  H.A.,  Proceedings  of  SAGEEP  '96,  April  1996. 
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NAEVA  •  Combined  Statistics:  Scenarios  1 ,2 


Detection  Statistics  (TMA*  Group) 


: i: 1 : : 

#  Baseline 

#  Detected 

Ordnance 

110 

103 

0.94  1  0.046  1 

Nonordnance 

127 

109 

0.86  1 

Total 

237 

212 

Number  False  Alarms 

202 

False  Alarm  Rate  (#/Hectare) 

24.82 

False  Alarm  Ratio  (#/Ord.) 

1.96 

Probability  False  Alarms 

0.031 

Locaiization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation  | 

dx  -  eastinq  error 

-0.28 

0.42 

dy  -  northing  error 

0.08 

0.42 

Radial  error 

0.58 

0.32 

iDepth  (z) _ 

dz  -  averaged  depth  error 

0.16 

0.24  I 

Idzl®-  absolute  depth  error 

0.28 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  |  #  Detected  Pd"  #  Correct  Pc® 

Ordnance 

110 

103 

0.94 

103 

1.00 

Nonordnance® 

128 

109 

0.85 

0 

0.00 

Ability  to  Size  _ I 

Large 

14 

14 

1.00 

11 

0.79 

Medium 

38 

34 

0.89 

29 

0.85 

Small 

58 

55 

0.95 

0 

0.00 

lAbilitv  to  Classify  I 

Bomb 

21 

20 

0.95 

20 

1.00 

Projectile 

41 

39 

0.95 

39 

1.00 

Mortar 

26 

26 

1.00 

0 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

18 

0.82 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

”  Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
“  Probability  of  correct  characterization 
^MA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 
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Depth  (meters) 


JPG  Controlled  Site:  Phase  ill 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator:  NAEVA 

Small  Medium  L 


■  Target  Detected 
□  Target  Not  Detected 
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13.0  OAO  CORPORATION^^ 

OAO  demonstrated  from  November  13  through  17,  1996  at  the  16-hectare  area  at  JPG. 
OAO  was  a  first  time  participant  in  the  UXO  ATD  program. 


13.1  TECHNOLOGY  DESCRIPTION 

13.1.1  System  Operational  Description 

The  TODS  remote  control  site  is  set  up  at  known  GPS  coordinates  near  to  suspect  UXO 
locations.  It  is  assumed  the  locations  are  documented  and  mapped  based  on  previous 
location  methods.  The  mapping  consist  of  either  markers  at  the  suspect  location  and/or 
GPS  coordinates  for  the  target.  Using  DGPS,  the  operator  drives  the  TODS  vehicle 
towards  target.  The  operator  then  either  excavates  the  target  based  upon  a  visual  marker 
or  uses  the  Schiebel  mine  detector  to  locate  the  UXO.  The  operator  would  then  use  the 
backhoe.  Air  Knife,  and  gripper  to  excavate  and  remediate  the  UXO. 


Test  Results  for  the  Teleoperated  Ordnance  Disposal  System  (TODS),  Phase  III 
UXO  Detection,  Identiflcation,  and  Remediation  Advanced  Technology 
Demonstration,  Jefferson  Proving  Ground,  Madison,  Indiana  - 1996 
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OAO  demonstrated  the  TODS  for  the  Phase  m  UXO  Detection,  Identification,  and 
Remediation  Advanced  Technology  Demonstration  at  Jefferson  Proving  Ground, 
Madison,  Indiana  in  November  1996.  A  40  acre  controlled  test  site  was  created  by 
emplacing  inert  ordnance  and  debris  at  documented  but  unpublished  locations.  The 
TODS  demonstrated  five  approaches  to  UXO  remediation  and  support  activities.  The 
first  and  simplest  involved  excavating  a  UXO  at  a  known  location.  This  proved  that  the 
TODS  was  an  effective  EOD  tool  under  remote  control  teleoperation.  The  second 
involved  navigating  with  GPS  and  excavating  a  marked  UXO.  This  proved  that  the  GPS 
could  be  used  to  navigate  the  TODS  until  visually  acquiring  the  target  and  then  proceed 
with  remediation.  The  third  scenario  involved  navigating  to  an  unmarked  target  using 
GPS,  acquiring  the  target  using  a  metal  detector  and  proceeding  with  excavation.  This 
proved  two  principles;  first,  that  GPS  was  accurate  to  get  within  1  meter  of  an  unmarked 
target,  and  second,  that  the  metal  detector  could  be  used  to  pinpoint  the  metallic  target 
within  the  GPS  error  zone.  The  fourth  demonstrated  approach  was  to  clear  vegetation 
from  an  area  under  teleoperation  to  allow  detection  or  surveying  activities.  The  fifth  was 
to  utilize  the  gripper  attachment  to  remove  exposed  UXO  from  excavated  holes  and 
handle  UXO  for  final  disposition.  Appendix  C  depicts  the  TODS  performing  various 
tasks  during  Ae  November  testing. 

Four  ordnance  scenarios  comprised  the  test  site  and  are  briefly  described.  The  Aerial 
Gunnery  Range  represented  aircraft  aerial  delivery  of  ordnance;  it  consisted  of  2.75  inch 
rockets  to  25  lb.  bombs  at  depths  from  near  surface  to  1.2  meters.  The  Artillery  and 
Mortar  Range  represented  conventional  ground  ordnance  fired  at  fixed,  hardened  targets; 
it  consisted  of  60-mm  mortars  to  8-inch  projectiles  at  depths  from  near  surface  to  1.2 
meters.  The  Grenades  and  Submunitions  Range  represented  a  conventional  impact  area 
delivered  by  aircraft  and  field  artillery;  it  contained  submunitions  and  grenades  at  depths 
shallower  than  0.5  meters.  The  Interrogation  and  Burial  Area  represented  a  conventional 
impact  area;  it  contained  2.75  inch  rockets  to  25  lb.  bombs  with  depths  from  near  surface 
to  1.2  meters. 

13.1.2  Remote  Controlled,  Teleoperated  Excavation 

The  TODS  is  based  upon  a  modified  New  Holland  L465  Skid-Steer  loader  that 
accommodates  a  modified  Woods  Backhoe  Arm.  The  skid  loader  and  backhoe  arm 
hydraulics  were  reconfigured  to  allow  remote  control  via  the  standard  OAO  remote 
control  electronics.  The  operator  commands  the  remote  vehicle  via  a  portable  control 
panel  with  switches  and  Joysticks.  The  vehicle  operates  as  a  manual  backhoe  would,  but 
the  operator  uses  two  vehicle  mounted  cameras  to  remotely  monitor  operations.  An  Air 
Knife  compressed  air  excavation  system  assists  in  non-intrusively  removing  dirt  and  mud 
while  excavating  and  identifying  UXO.  Two  common  uses  are  when  seeking  first  contact 
with  the  buried  UXO,  and  searching  the  over  burden  pile. 
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13.1.3  Metal  Detection 


A  Schiebel  AN- 19/2  Mine  Detector  was  modified  to  allow  its  remote  operation  on 
TODS.  This  included  an  interface  to  allow  the  detection  to  be  observed  by  the  operator 
via  video  overlay,  and  an  actuator  that  allows  the  operator  to  remotely  raise  and  lower  the 
detection  head.  The  detector  was  used  to  acquire  unmarked  targets,  to  reacquire  targets 
during  the  excavations  and  to  locate  targets  that  were  excavated  into  the  overburden  pile. 
The  detector  was  also  used  to  establish  the  UXO's  geometry  for  planning  the  TODS’ 
approach  (backhoe  arm  perpendicular  to  the  linear  axis  of  the  UXO).  The  detector's 
sensitivity  was  manually  set  prior  to  a  mission  when  approximate  depths  were  known  by 
holding  a  metallic  object  the  known  distance  from  the  detection  head  and  manually 
adjusting  the  Schiebel's  sensitivity  knob  control. 

13.1.4  Global  Positioning  System 

The  TODS  utilized  a  Differential  Global  Positioning  System  (DGPS)  to  guide  the  remote 
controlled  vehicle  to  the  predetermined  UXO  locations  with  the  Base  Station  antenna  at  a 
GPS  monument.  The  DGPS  provided  the  operator  the  location  of  the  vehicle.  The  TODS 
proved  to  be^  able  to  provide  positions  in  real  time  with  a  nominal  accuracy  within  4 
meters,  and  a'lO  minute  occupancy  accuracy  near  1  meter. 

13.1.5  Vegetation  Clearance 

To  clear  vegetation  prior  to  UXO  detection  or  surveying  activity,  the  TODS  allows  the 
attachment  of  a  rotary  cutter  (instead  of  the  backhoe  arm).  The  cutter  is  COTS  with  a 
standard  mounting  attachment  for  fast  and  simple  installation  and  removal.  The  cutter 
has  a  55  in.  diameter  cutting  capacity  and  can  cut  vegetation  up  to  3  inches  in  diameter. 
The  operator  remotely  commands  the  cutter  on/off  as  well  as  the  height  and  angle  of  the 
cutter  via  the  portable  controller.  Blade  speed  is  a  function  of  the  vehicle  throttle. 

13.1.6  Manipulation 

When  grasping,  rather  than  digging  is  required,  the  operator  can  easily  reconfigure  the 
arm  in  the  field  to,  in  place  of  the  bucket,  attach  the  one  degree  of  freedom  gripper  to 
hold  and  manipulate  UXO.  The  gripper  utilizes  a  dual  piston,  self-centering  mechanism 
to  grasp  objects  up  to  10  inches  in  diameter.  The  gripping  force  can  be  manually  set  prior 
to  the  mission  from  negligible  to  1300  lb. 

13.2  DEMONSTRATION  RESULTS 

The  TODS  successfully  excavated  18  targets.  2  targets  were  used  to  demonstrate 
navigation  only  and  were  not  excavated.  Four  more  targets  were  known  anomalies  for  a 
total  of  twenty-four  targets.  The  average  excavation  time  was  27  minutes.  The  average 
travel  rate  was  2  km/h.  10  targets  were  successfully  remediated  from  marked  locations 
with  no  navigational  requirement.  OAO  successfully  located  10  targets  using  GPS 
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navigation,  6  were  to  marked  locations  and  4  were  to  unmarked  locations.  8  of  the  10 
GPS  navigations  were  excavated;  2  of  the  GPS  navigations  were  to  only  acquire 
unmarked  target  locations.  A  surveying  crew  quantified  the  resultant  accuracy.  The  first 
target  was  located  within  less  than  0.5  meters  of  its  actual  location  and  the  second  was 
located  within  5  cm.  Using  the  metal  detector  was  successful  in  locating  all  4  unmarked 
targets.  Sweeps  conducted  by  the  metal  detector  at  unmarked  locations  encompassed  4 
meter  squares.  The  GPS  subsystem  proved  to  be  accurate  within  a  2  meter  square  with  a 
precision  of  better  than  1  meter.  The  TODS  also  demonstrated  vegetation  cutting  as  is 
required  to  place  surveyor's  marks  and  demonstrated  the  gripper  operation  to  assist  in 
grasping  some  UXO  in  the  excavated  holes  for  final  disposition. 

13.2.1  Assumptions 

It  was  assumed  that  all  points  in  the  16  hectare  area  would  be  visible  from  at  least  one 
location  so  that  line-of-sight  was  available  for  video  signal  reception.  This  was  found  not 
to  be  true  and  the  OAO  team  set  up  a  mobile  Base  Station  utilizing  a  mini-van  and  a 
portable  generator  for  power.  Each  reconfiguration  required  that  the  TODS  GPS  Base 
Station  was  set  up  on  a  known  GPS  monument.  It  was  assumed  that  target  GPS 
coordinates  provided  by  PRC  were  within  2  cm  in  any  horizontal  direction  of  the  actual 
target  locatioil.  Two  or  more  targets  were  not  to  be  located  within  2  meters  of  each  other. 
It  was  further  assumed  that  it  would  be  acceptable  to  transport  UXO  to  their  final 
destination  in  the  bucket  as  convenient  throughout  the  day.  In  the  event  that  gripper 
operations  are  require,  it  would  be  acceptable  to  excavate  numerous  UXO,  reconfigure 
with  grippers,  and  return  to  transport  the  multiple  UXO  to  their  final  destination.  Due  to 
the  physical  limitations  of  the  backhoe  arm's  geometry,  it  was  assumed  that  no  targets 
would  be  excavated  deeper  than  1.2  meters. 

13.2.2  Site-Specific  Procedures 

Procedures  specific  to  JPG  included  mobile  capability  of  the  Base  Station,  hazardous 
terrain  avoidance  and  Schiebel  detector  calibration.  Due  to  the  numbers  of  trees  and  the 
rolling  terrain,  and  the  required  line  of  sight  for  video  transmission,  the  Base  Station 
could  not  be  placed  at  any  single  location  to  cover  all  four  ranges.  Sufficient,  surveyed 
GPS  monuments  were  available,  however,  to  accommodate  range  specific  Base  Station 
locations.  A  generator  was  required  to  power  the  mobile  Base  Station,  and  refueling 
was  required  several  times  a  day.  A  mini-van  was  large  enough  to  accommodate  the 
remote  control  equipment  as  well  as  the  operator,  JPG  test  observer  and  data  recorder. 
All  antennas  and  stands  were  easily  assembled  and  disassembled  for  each  move. 

Several  times  during  the  test,  the  Schiebel  metal  detector's  sensitivity  was  recalibrated 
based  upon  provided  knowledge  of  the  depth  of  targets.  During  the  GPS  navigations 
mode,  it  was  often  required  to  move  an  exact  number  of  meters  in  the  latitude  or 
longitudinal  axes.  By  observing  the  wheel  and  counting  revolutions,  the  operator  could 
drive  the  TODS  an  exact  distance  in  an  axis. 
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The  poor  drainage  throughout  the  JPG  test  site  created  puddles,  mud  holes  and  other 
slippery  conditions  causing  poor  traction.  This  never  caused  a  mission  failure  but  was 
more  a  matter  of  inconvenience  and  required  additional  planning.  This  was  mostly  an 
issue  during  mine  detection  sweeps  when  skid  steering  tore  up  the  ground  cover  removing 
the  last  vestige  of  traction.  At  no  time  during  the  test  did  the  vehicle  require  ancillary 
power  to  traverse  the  test  site. 

No  problems  were  encountered  although  enhancements  are  targeted  to  improve  vehicle 
and  mission  efficiency.  To  reduce  the  travel  time  during  GPS  navigations,  quicker 
vehicle  position  updates  would  be  required.  In  addition,  a  vehicle  mounted  compass 
would  assist  the  operator  in  determining  the  direction  of  travel  desired.  Combining  these 
two  enhancements  would  allow  for  a  more  direct  route  traveled  between  targets.  A 
calculation  by  the  operator  could  be  performed  to  quickly  determine  the  distance  and 
direction  of  the  next  target.  The  portable  controller  will  be  modified  to  reflect  final 
configuration.  These  modifications  will  include  joystick  control  of  the  backhoe  arm  to 
more  closely  follow  the  controls  of  a  manual  backhoe.  This  will  human  engineer  the 
operator  interface  and  allow  the  TODS  operation  similar  to  manual  backhoe  operation 
with  minimal  training. 

A  remote  adjust  capability  for  the  metal  detector  sensitivity  would  be  desirable  to  aid  in 
the  detection  of  multiple  metal  objects  at  varying  burial  depths  during  a  single  mission. 
During  practical  field  use  it  was  discovered  that  the  remote  operator  needs  to  switch  from 
detection  of  deeply  buried  objects  to  detection  of  shallow  buried  objects  such  as  the 
investigation  of  excavated  overburden  piles. 

Depth  perception  for  the  vision  system  would  greatly  enhance  the  operation.  Depth 
perception  will  allow  the  remote  operator  better  visual  sensitivity  with  UXO  discovery, 
identification,  and  removal.  In  addition,  OAO  recommends  accompanying  the  excavator 
with  a  smaller,  highly  visible,  remote  observing  platforms  allowing  for  viewing  the 
excavation  progress  from  many  different  angles. 

13.2.3  Discussion  of  Resuits 

The  24  missions  conducted  by  OAO  personnel  with  the  TODS  at  Jefferson  Proving 
Ground  were  over  twice  in  quantity  above  the  proposed  1 1  missions.  14  excavations  at 
marked  targets  with  no  navigational  requirements  were  conducted  where  4  were 
proposed.  10  of  these  resulted  in  remediation  of  the  targeted  UXO.  4  excavations  were 
conducted  at  locations  predetermined  as  anomalies  where  nothing  was  found.  1  anomaly 
target  was  beyond  the  extent  of  the  system  limitations  as  described  in  the  assumptions 
and  PRC  limited  the  excavation  time.  6  GPS  navigations  to  marked  UXO  locations  were 
conducted  where  3  were  proposed.  Successful  excavations  and  UXO  removals  were 
conducted  at  each  target.  4  GPS  navigations  to  unmarked  UXO  locations  were 
conducted  as  was  proposed.  Final  determination  of  each  target  was  pin-pointed  by  an 
organized  pattern  sweep  with  a  metal  detector.  In  the  first  2  cases,  OAO  personnel 
informed  JPG  test  personnel  of  the  target  location  as  determined  by  the  center  of  the 


B.13-5 


detectors  search  head.  An  on-site  surveying  crew  then  compared  the  actual  location  to 
the  estimated.  In  both  cases,  the  TODS  was  close  enough  that  successful  excavations 
would  have  resulted  had  JPG  personnel  requested  them.  The  last  2  UXO  were  excavated 
successfully  given  only  the  GPS  coordinates,  the  approximate  size  and  the  approximate 
burial  depth. 

The  average  excavation  time  across  all  scenarios  was  27  minutes.  Excavation  times  can 
be  increased  dramatically  with  operator  training  and  UXO  experience,  as  was  evident  by 
the  OAO  operator  improving  his  dig  times  as  the  test  progressed.  On  the  final  day, 
excavation  times  were  reduced  to  13,  17  and  8  minutes.  The  last  excavation  (target 
1850),  the  operator  demonstrated  the  delicacy  of  the  backhoe  arm  by  completely 
uncovering  a  30"  rocket  before  removing  it  where  inexperience  resulted  in  damaging  a 
rocket  earlier  (target  1852).  The  removal  time  for  the  last  target  was  24  minutes. 

The  metal  detector  was  used  effectively  to  both  determine  initial  target  locations  and  also 
to  reinvestigate  excavation  holes  and  overburden  piles  to  reacquire  targets.  OAO 
successfully  located  four  unmarked  targets  with  the  metal  detector.  Unbeknownst  to  the 
OAO  operator  at  the  start  of  the  test,  there  are  many  schemes  that  EOD  personnel  use 
with  the  detectors  in  the  determination  of  UXO  type,  size,  depth  and  orientation.  EOD 
personnel  use^  learned  techniques  with  the  metal  detector  to  facilitate  in  organizing  and 
expediting  an  excavation.  OAO  personnel  were  given  such  hints  piece-meal  as  the  test 
progressed. 

The  average  travel  rate  was  2  km/h.  The  travel  rate  will  increase  with  the  next  phase  of 
improvements  to  the  TODS.  Some  time  was  spent  avoiding  hazardous  terrain.  Yet,  most 
of  the  excess  time  was  due  to  the  current  GPS  subsystem  settling  time  which  will  be 
eliminated  in  the  next  phase  of  the  TODS. 

The  vegetation  cutter  was  used  to  trim  grass  three  feet  high  with  sparse  populations  of 
small  shrubs.  This  is  beneficial  for  practical  applications  where  it  is  not  advisable  to  send 
in  manned  mowers. 

The  grippers  were  demonstrated  to  show  the  ability  to  retrieve  UXO  or  fragments  when 
using  the  bucket  is  not  desirable.  In  one  instance,  the  TODS  placed  the  UXO  in  the  back 
of  a  van  as  the  final  disposition. 

13.2.4.  Data 


13.2.4.1  Raw  Data 

The  raw  data  is  attached  as  an  Appendix  A  in  both  hard  and  soft  copies. 

13.2.4.2  Processed  Data 
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Table  1  presents  data  concerning  the  TODS  JPG  test.  The  supporting  derived  data  is 
attached  as  Appendix  B. 

13.2.5  Conclusions 

The  TODS  proved  to  be  a  quick,  efficient  and  cost  effective  system  for  the  remediation  of 
UXO.  The  TODS  average  travel  time  using  GPS  was  2  km/hr  but  in  the  next  phase  of 
TODS,  will  increase  closer  to  the  10  km/hr  system  velocity.  The  average  excavation  time 
was  0.45  hours  per  hole.  These  statistics  along  with  the  100  percent  success  rate 
outperformed  previous  JPG  and  Live  Site  Technology  demonstrators.  These  times  will 
improve  based  upon  enhancements  discussed  as  well  as  training  and  the  use  of  EOD- 
knowledgeable  personnel  to  operate  TODS.  As  was  discussed  excavation  times  could  be 
reduced  by  being  more  familiar  with  the  use  of  the  detector,  excavation  planning,  and 
excavation  techniques.  The  TODS  proved  to  be  extremely  reliable  with  no  breakdowns 
suffered  and  the  ability  to  work  eight  hour  days  consistently  demonstrated.  The  OAO 
crew  required  two  people  to  operate  the  system;  an  operator  and  a  GPS  navigator.  Again, 
enhancements  discussed  can  eliminate  the  GPS  need  and  result  in  a  single  person  to 
effectively  operate  the  system.  The  ability  to  work  8  hour  days  without  tiring  results  in 
further  efficiencies  and  can  be  taken  advantage  of  by  trading  off  EOD  operators 
throughout  a  working  day.  In  addition,  the  ability  to  operate  the  system  from  a  controlled 
environment  eliminated  environmental  impacts  that  typically  result  in  operator  fatigue. 
The  TODS  worked  in  the  cold  and  the  rain  while  the  operators  sat  in  a  warm,  dry  van. 
The  TODS  required  minimal  setup  with  a  single  non-technical  person  able  to  accomplish 
this  activity  in  less  than  0.5  hours.  The  TODS  cost  is  about  $200k  with  many  of  the 
components  being  required  to  conduct  EOD  work  manually  such  as  the  skid  loader  with 
backhoe,  GPS,  and  mine  detector.  This  is  a  minimal  investment  compared  to  what  would 
be  manually  required.  The  demonstrated  efficiencies,  speed  of  remediation,  and  inherent 
safety  prove  the  TODS  to  be  a  viable  EOD  tool  for  UXO  remediation. 
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14.0  ROCKWELL^ 

Rockwell  demonstrated  from  October  2  through  5,  1996  at  the  16-hectare  area  at  JPG. 
Rockwell  was  a  first  time  participant  in  the  UXO  ATD  program. 

14.1  TECHNOLOGY  DESCRIPTION 


14.1.1  Sensor  System  and  Transport  Mode 

The  sensor  system  is  a  new  high-performance  magnetic  search  system  design  especially 
for  UXO  surveys  and  for  surveys  having  similar  requirements.  The  complete  system 
consists  of  four  main  components:  a  sensor  subsystem,  a  position  monitoring  subsystem, 
a  data  handling  subsystem,  and  a  carry  assembly. 


ROCKWELL,  Jefferson  Proving  Ground,  UXO  Technology  Demonstration, 
Demonstration  Summary  Report  -  November  4, 1996 
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The  sensor  subsystem  is  a  four  foot  array  of  3-axis  fluxgate  magnetometers  arranged  in  ^ 
in-line  differential  configuration.  Magnetic  “balance”  among  the  magnetometers  is 
maintained  by  the  rigidity  of  the  housing  design  and  a  proprietary  imbalance 
compensation  mechanism.  The  position  monitoring  subsystem  provides  accurate  three- 
dimensional  position  information  for  all  magnetometers  in  the  sensor  unit.  This  is 
accomplished  with  a  commercially  available  carrier  phase  tracking  differential  GPS 
system  (Trimble  model  7400MSi)  and  a  commercially  available  accelerometer  (Applied 
Physics  Systems).  The  GPS  system  provides  accurate  position  information  and  the 
accelerometer,  in  conjunction  with  the  magnetometer,  provides  attitude  information. 
These  are  combined  in  post-processing  to  determine  magnetometer  positions  at  each  data 
sample,  based  on  the  known  geometry  of  the  system.  Both  the  position  information  and 
the  attitude  information  are  recorded  by  the  data  handling  subsystem,  and  are  time-keyed 
to  the  magnetometer  data  for  proper  recovery  during  post-processing.  The  data  handling 
system  consists  of  proprietary  electronics  coupled  to  a  commercially  available  notebook 
computer  (Gateway  2000  133  MHz  Solo  with  a  1.4  GByte  removable  hard  drive)  for 
control  and  recording  of  the  data.  The  carry  assembly  is  a  specially  designed  pair  of 
backpacks.  The  system  is  currently  configured  to  be  carried  by  two  people  in  a  leader- 
follower  arrangement.  The  lead  unit  has  a  special  cradle  assembly  for  the  sensor 
subsystem,  a^d  also  contains  the  accelerometer  and  the  GPS  antenna.  The  carry  height 
for  the  sensors  is  about  1  m  above  the  ground,  yielding  an  appropriate  balance  of 
performance  and  ergonomics.  The  back  unit  contains  the  power  supply,  a  daylight 
display,  the  notebook  computer,  and  much  of  the  electronics.  The  two  units  are 
connected  by  an  electrical  umbilical. 

This  system,  along  with  proprietary  data  processing  algorithms  and  software,  yields  a 
swath  width  substantially  wider  than  the  sensor  array  itself.  This  in  turn  yields  higher 
areal  coverage  rates  and  faster  more  economical  surveys.  In  addition,  it  provides  a  stand¬ 
off  range  for  detecting  and  locating  otherwise  inaccessible  targets. 

14.1.2  Recommended  Applications  and  Technology  Limitations 

The  system  is  a  very  sensitive  passive  magnetic  device  and  is  capable  of  detecting 
anomalies  in  the  ambient  magnetic  field.  Most  frequently  such  anomalies  are  due  to 
concentrations  of  ferrous  material  in  an  otherwise  non-ferrous  environment.  Because  of 
the  unique  design  of  the  system,  it  is  able  to  not  only  detect  such  anomalies,  but  it  can 
accurately  estimate  their  locations  in  three-dimensions  and  their  magnetic  strengths.  The 
system  can  be  used  to  survey  any  terrain  which  is  traversable  by  two  people  carrying 
moderate  weight  backpacks.  The  search  strategy  for  the  area  can  be  easily  tailored  to  the 
specific  terrain  and  conditions  of  the  site.  Grid  layouts  or  special  paths  are  not  required. 
In  areas  where  GPS  does  not  perform  well  because  of  tree  canopy  or  other  obstructions, 
the  system  can  be  augmented  by  ultrasonic  or  optical  positioning  systems.  Except  for 
thunderstorms,  the  system  can  be  used  in  most  weather  conditions.  For  extreme 
conditions  of  rain,  humidity,  blowing  dust  or  sand,  the  system  would  be  repackaged  to 
protect  internal  components.  In  conditions  of  extreme  heat  or  cold  the  system  may 
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require  site-specific  calibration  procedures,  depending  on  the  level  of  performance 
desired. 

The  current  system  is  designed  to  be  man-portable,  but  could  be  readily  configured  for 
vehicle  mounting  (towed,  pushed,  or  beam-mounted).  It  could  also  be  configured  for  use 
by  low-flying  remotely  piloted  aircraft.  In  addition,  it  could  be  packaged  for  underwater 
use,  either  in  a  hand-held  arrangement  or  in  a  towed,  pushed,  or  beam-mounted  system. 

14.1.3  Logistics  Requirements 

The  system  is  small  enough  to  be  transported  in  a  mini- van  or  a  small  truck,  and  is  readily 
shipped  over  large  distances  by  a  commercial  air  carrier  (as  was  done  for  this 
demonstration).  The  system  is  powered  by  batteries,  which  must  be  recharged  and 
replaced  on  a  regular  basis.  For  best  performance,  an  on-site  calibration  of  the  system  can 
be  performed  using  a  specially  designed  calibration  stand. 

14.1.4  Data  Acquisition 

The  system  collects  and  stores  data  at  a  sample  rate  that  results  in  oversampling  by  at 
least  a  factor'^of  two  for  normal  walking  speeds.  The  sample  rate  can  be  increased  for 
higher  speed  vehicle  mounted  searches.  The  stored  data  includes  magnetometer, 
differential  magnetometer,  magnetic  imbalance  compensation,  accelerometer,  and  GPS 
Hata  All  data  is  initially  recorded  on  a  removable  hard  drive  in  the  system’s  notebook 
computer.  Periodically  the  data  is  compressed,  and  transferred  to  diskettes  for  back-up, 
data  quality  checks,  and  for  processing. 

14.1.5  Data  processing  and  Interpretation 

After  the  survey  is  completed  all  data  is  transferred  to  a  desktop  personal  computer  and 
decompressed.  Rockwell  proprietary  software  is  then  used  to  process  the  data  using  two 
desktop  computers  (133  MHz  Pentiums  with  32  MBytes  RAM  and  800  MByte  hard 
drives)  with  a  shared  1  GByte  directory  on  a  LAN. 

It  was  originally  planned  to  process  the  data  in  three  stages  as  follows.  First,  a  number  of 
corrections  are  applied  to  the  magnetic  sensor  data  in  order  to  recover  accurate  values  for 
the  differential  magnetic  field  signals.  The  corrections  include  electronic  gains,  sensor 
calibrations,  mismatch  and  rotational  imbalance  corrections,  and  attitude  corrections. 
Second,  the  corrected  data  are  then  processed  to  recover  the  sources  responsible  for 
producing  the  observed  magnetic  field  data.  This  is  done  in  a  marmer  equivalent  to  a 
least  squares  fit  to  the  data.  The  sources  are  assumed  to  belong  to  three  categories: 
geology,  line-like  sources,  and  dipole-like  sources.  Pertinent  parameters  are  determined 
for  each  source.  For  the  dipole-like  sources,  the  parameters  are  3-D  location,  magnetic 
moment,  and  moment  orientation.  Finally,  for  each  dipole-like  source  a  determination  is 
made  as  to  whether  or  not  the  source  is  an  object  of  interest.  This  is  done  on  the  basis  of 
the  size  of  the  source’s  magnetic  moment,  its  orientation,  and  its  depth. 
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The  preceding  paragraph  describes  the  data  processing  that  was  planned  for  this 
demonstration.  However,  the  system  experienced  an  electronic  failure  during  the  survey, 
and  during  post-processing  some  other  system  problems  were  uncovered  (see  section 
3.3).  As  a  consequence  the  planned  data  processing  procedure  outlined  above  was 
amended.  Some  additional  filtering  was  performed  on  the  data  and  a  less  ambitious 
approach  to  characterizing  the  targets  was  used.  The  strength  estimates  were  converted  to 
target  weight  estimates  using  a  simple  proportionality  between  moment  strength  and 
weight  for  typical  ferrous  objects  in  the  earth’s  field.  This  proportionality  is  usually  good 
to  within  a  factor  of  3,  but  may  be  off  by  as  much  as  a  factor  of  10  in  some  circumstances. 
Nevertheless,  it  provides  a  mechanism  for  grouping  the  targets  into  small,  medium,  or 
large  weight  categories.  The  performance  level  of  the  fully  functioning  system  would  be 
adequate  to  permit  further  interpretation  of  the  data,  especially  in  conjunction  with  an 
appropriate  target  database. 

14.1.6  Quality  Assurance 

During  a  survey,  the  quality  of  the  data  is  checked  at  regular  intervals.  To  accomplish 
this,  survey  data  (stored  on  the  hard  disk  of  the  system  notebook  computer)  is  transferred 
to  another  cJimputer.  Randomly  selected  portions  of  the  data  are  then  plotted  and 
examined  in  order  to  assess  its  quality.  For  best  performance  the  system  is  also  calibrated 
both  before  and  after  the  survey,  using  a  special  calibration  stand.  The  stand  permits 
controlled  orientation  of  the  system  within  the  ambient  field.  All  aspects  of  the  system 
can  be  calibrated  in  this  way,  except  for  an  overall  scale  factor  for  the  magnetometers, 
which  must  be  established  in  advance  by  special  calibration  techniques  employing  a 
Helmholtz  coil  facility  at  A&MSD’s  facility  in  Anaheim,  California. 

14.2  DEMONSTRATION  RESULTS 

14.2.1  Assumptions 

Only  three  assumptions  were  made  in  planning  and  executing  the  survey.  First,  it  was 
assumed  that  in  the  data  processing  all  sources  of  interest  in  the  survey  areas  could  be 
represented  as  magnetic  dipoles  at  depths  of  0  to  12  m.  In  particular,  this  excludes  non- 
ferrous  objects  such  as  aluminum  rocket  casings  in  the  AGR.  (The  AGR  became  the 
replacement  area  for  A&MSD  in  lieu  of  the  second  artillery  and  mortar  range  requested  in 
Rockwell’s  original  proposal).  Second,  it  was  assumed  that  the  terrain  would  permit  a 
walking  speed  of  at  least  1  m/s.  Third,  it  was  assumed  that  carrier  phase  tracking 
differential  GPS  would  be  functional  throughout  most  of  the  survey  areas. 

14.2.2  Site-Specific  Procedures 

The  amended  proposal  called  for  surveying  both  the  AMR  (scenario  2)  and  the  AGR 
(scenario  1)  in  32  hours.  A  preplanned  search  strategy  was  adopted  to  efficiently  cover 
the  two  survey  areas.  The  plan  called  for  following  straight  line  paths  where  terrain  and 
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vegetation  permitted,  and  using  paths  of  opportunity  otherwise.  A  1.5  m  path  separation 
was  adopted,  which  resulted  in  146  paths  for  the  AMR  and  122  paths  for  the  AGR.  The 
swath  width  of  the  system  for  the  smallest  targets  of  interest  was  expected  to  be  about  2.3 
m,  so  that  the  1.5  m  path  separation  provided  a  conservative  overlap  of  nearly  70%. 

The  system  was  carried  by  three  two-man  teams  in  rotation.  Special  path-following 
software  and  a  simple  daylight  display  permitted  the  carriers  to  follow  the  preplanned 
paths  without  the  need  for  grid  layouts  or  other  path  markers.  Except  for  walking  around 
isolated  trees  and  some  deeper  standing  water  areas  the  survey  teams  were  able  to  follow 
the  preplanned  search  strategy  over  most  of  both  sites.  There  were  three  areas,  however, 
where  standing  and  fallen  trees,  underbrush,  and  bramble  did  not  permit  following  the 
preplanned  paths:  1)  a  small  area  along  the  north  central  boundary  of  the  AMR,  2)  a  much 
larger  area  near  the  west  central  boundary  of  the  AMR  and  a  modest  sized  area  in  the 
southeast  comer  of  the  AGR.  All  of  these  areas  were  surveyed  following  paths  of 
opportunities. 

At  the  assumed  minimal  walking  speed  of  1  m/s  the  AMR  and  the  AGR  could  both  be 
covered  in  about  25  hours,  leaving  a  safety  margin  of  7  hours  in  the  allotted  32  hour 
performance  period.  The  two  areas  were  acmally  covered  at  a  significantly  higher  rate, 
finishing  in  about  21  hours  of  total  survey  time.  This  left  time  for  a  system  re-calibration 
and  a  rerun  over  the  Demonstration  Reference  Area  (DRA)  within  the  allotted 
performance  time,  despite  a  24  hour  delay  due  to  an  electronics  part  failure  (see  section 
3.3). 

14.2.3  Problems  Encountered 

A  battery  harness  was  inadvertently  connected  backwards  early  on  the  morning  of  the 
second  day  (Thursday)  causing  the  harness  to  bum  out.  A  replacement  harness  was 
installed,  permitting  surveying  to  continue.  A  simple  change  in  the  battery  harness  design 
will  prevent  a  future  recurrence  of  this  problem.  Later  the  same  day  an  electronics  failure 
occurred  rendering  the  system  inoperable.  Cause  of  the  failure  is  not  known,  but  may 
have  resulted  from  stress  on  the  electronics  due  to  the  earlier  battery  harness  episode. 
The  system  was  repaired  late  Friday  evening  with  parts  flown  in  from  California.  The 
repair  was  successful  in  getting  the  main  sensor  channels  operable,  but  left  some 
imbalance  compensation  channels  non-functional.  The  system  was  used  in  this  somewhat 
degraded  state  to  finish  the  survey  on  Saturday  within  the  allotted  time  frame. 

Occasional  degradation  of  GPS  performance  occurred  during  the  surveys,  presumably 
because  of  satellite  blockage  by  trees.  This  caused  the  GPS  accuracy  level  to  drop  from  a 
few  centimeters  to  a  meter  (or  worse,  in  a  few  instances).  This  in  turn  tended  to  degrade 
overall  system  performance  in  the  sense  of  increasing  target  location  and  moment  errors. 

During  post-processing  of  the  data  it  was  discovered  that  there  were  a  few  previously 
unnoticed  problems  with  the  system.  From  a  system  stand-point  these  problems  are 
easily  corrected  and  will  be  done  so  as  a  part  of  the  on-going  development  process.  First, 
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there  appeared  to  be  magnetic  sources  internal  to  the  system  which  were  contributing  to 
the  noise  level.  Two  such  sources  were  subsequently  found.  One  was  a  piece  of  GPS 
equipment  in  the  front  backpack  and  the  other  involved  some  special  alignment  pins 
within  the  sensor  subsystem.  Second,  the  imbalance  compensation  mechanism  had 
apparently  failed  earlier  than  previously  recognized,  so  that  the  full  capability  of  the 
imbalance  compensation  was  not  available  throughout  most  of  the  survey.  Third,  it  was 
determined  that  the  front  backpack  frame  is  not  sufficiently  rigid,  allowing  the  sensor  unit 
to  sag  about  4.5  degrees  from  its  position  in  the  calibration  stand.  As  far  as  the  survey 
data  is  concerned,  special  filtering  and  re-calibration  procedures  were  implemented  to 
partially  compensate  for  these  problems. 

14.2.4  Discussion  of  Results 

The  noise  level  for  the  system  in  operation  on  the  two  sites  was  0.73  nT  rms  per 
difference  channel.  This  is  somewhat  higher  than  expected  due  to  the  problems  discussed 
in  section  3.3.  However,  it  is  still  a  very  respectable  operational  noise  level  for  a  passive 
magnetic  system.  With  minor  improvements  and  corrections,  the  system  noise  level  can 
be  readily  reduced  to  the  0.1  nT  target  level.  It  was  originally  planned  to  use  a  detection 
threshold  of  15  dB  above  the  noise  level.  With  that  detection  threshold  many  individual 
targets  are  detected  multiple  times  on  adjacent  search  paths.  The  original  processing  plan 
called  for  grouping  these  multiple  detections  together  and  processing  them  as  a  whole  to 
recover  target  features.  Because  of  the  slightly  degraded  performance,  it  was  deemed 
better  to  increase  the  threshold  to  18  dB  and  eliminate  the  weaker  (more  corrupted) 
multiple  detections  of  each  target.  This  eliminated  59%  of  the  individual  detections, 
some  of  which  may  have  been  singly  detected  weak  targets.  Of  the  remaining  detections 
about  32%  were  determined  to  represent  the  same  targets  and  were  grouped  accordingly. 

The  AMR  results  exhibit  a  clear  line  of  magnetic  sources  roughly  parallel  to  the  western 
boundary  and  passing  about  16  m  to  the  east  of  monument  #3.  The  line  coincides  with  a 
slight  rise  in  the  southwest  comer  of  the  site  and  passes  thorough  the  wooded  and 
brambly  area  2)  mentioned  in  section  3.2.  This  may  be  a  fence  row,  a  fragmented 
pipeline,  geology,  or  a  collection  of  interesting  targets.  The  targets  seem  to  be  well 
isolated  and  hence  may,  in  fact,  be  targets  of  interest. 

14.2.5  Raw  Data 

The  raw  data  was  compressed  and  stored  on  39  high  density  diskettes.  Total  amount  of 
decompressed  raw  data  is  nearly  90  MBytes.  It  consists  of  magnetometer,  differential 
magnetometer,  imbalance  compensation,  accelerometer,  and  GPS  data. 

14.2.6  Processed  Data 

The  total  amount  of  stored  data  at  all  stages  of  processing  is  in  excess  of  600  MBytes.  It 
includes  data  to  show  the  paths  actually  followed  during  the  survey,  system  attitude, 
transformed  and  filtered  signal  data,  differential  field  contour  maps,  threshold  statistics. 
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detection  groupings,  target  locations  and  target  strengths.  Details  of  the  target  features 
resulting  form  the  data  processing  are  provided  in  the  attached  data  package  using  the 
prescribed  data  format. 

14.2.7  Conclusions 

The  subcontract  agreement  called  for  surveying  two  10  acre  scenarios  in  a  32  hour  time 
period:  the  Artillery  and  Mortar  Range  (scenario  2)  and  the  Aerial  Gunnery  Range 
(scenario  1).  The  original  proposal  called  for  surveying  two  distinct  artillery  and  mortar 
ranges,  but  because  the  second  such  range  was  not  available  the  aerial  gunnery  range  was 
substituted  in  its  stead.  Both  sites  were  completely  surveyed  within  the  allotted  time 
period.  In  fact,  the  actual  survey  time  on  the  “grid”  was  about  21  hours.  Some  system 
electronics  problems  occurred  which  slightly  degraded  system  performance,  but  the  level 
of  performance  was  still  very  respectable.  All  data  was  successfully  recorded  and 
processed  and  the  results  reported  in  the  attached  data  package  using  the  prescribed 
format. 
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Rockwell  •  Combined  Statistics:  Scenarios  1  and  2 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

#  Detected 

Pd° 

Ordnance 

110 

37 

0.34 

0.038  1 

Nonordnance 

127 

35 

0.28 

1 

Total 

237 

72 

Number  False  Alarms 

211 

False  Alarm  Rate  {#/Hectare) 

25.93 

False  Alarm  Ratio  (#/Ord.) 

5.70 

Probability  False  Alarms 

0.033 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Position  (x.v) 

dx  -  easting  error 

0.02 

0.43 

dv  -  northing  error 

0.06 

0.74 

Radiai  error 

0.71 

0.48 

Depth  (2) _ J 

dz  -  averaged  depth  error 

0.04 

0.27 1| 

Idzl'-  absolute  depth  error 

0.26 

Characterization  Statistics  (TMA  Closest) 


1 

#  Baseline 

#  Detected 

Pd" 

#  Correct 

Pc'- 

Ordnance 

110 

37 

0.34 

0 

0.00 

Nonordnance® 

128 

35 

0.27 

0 

0.00 

Large 

14 

12 

0.86 

0 

0.00 

Medium 

38 

13 

0.34 

0 

0.00 

Smali 

58 

12 

0.21 

0 

0.00 

Ability  to  Classify 

Bomb 

21 

17 

0.81 

0 

0.00 

Proiectile 

41 

12 

0.29 

0 

0.00 

Mortar 

26 

2 

0.08 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

22 

6 

0.27 

0 

0.00 

Notes:  ‘  Target  Matching  Algorithm 

^  Probability  of  detection 
’’ Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
®TMA  affects  how  nonordnance  baseline  targets  are  counted 
NA  •  not  appiicable 
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Depth  (meters) 
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15.0  SANFORD  COHEN  AND  ASSOCIATES  (SC&A)^^ 

Sanford  Cohen  and  Associates  (SC&A)  analyzed  data  collected  by  three  participants  in 
the  JPG  m  Advanced  Technology  Demonstration  using  advanced  Ordnance  and 
Explosives  Knowledgebase  (OE-KB)  concepts.  The  participating  companies  are:  ADI 
Limited,  GeoCenters,  Inc.,  and  Geometries.  The  objectives  of  this  analysis  are: 

1.  Demonstrate  the  effectiveness  of  OE-KB  techniques  when  applied  to  industry  standard 
geophysical  data  sets  as  compared  to  other  techniques  demonstrated  by  the  data  suppliers. 

2.  Isolate  performance  differences  in  geophysics  instruments  and  survey  methodology 
by  providing  a  baseline  anSysis  technique  applied  identicSly  to  three  independent  data 
sets. 

3.  Demonstrate  effective  coordination  between  companies  in  the  OE  detection  and 
discrimination  market. 

Information  on  system  transport,  sensor  type,  navigation  systems  and  survey  techniques 
can  be  found  in  the  sections  provided  by  the  participating  companies  and  is  not  addressed 
here. 

15.1  TECHNOLOGY  DESCRIPTION 

The  OE-KB  techniques  have  been  developed  by  SC4feA  under  the  sponsorship  of  the  US 
Army  Corps  of  Engineers,  Engineering  and  Support  Center,  Huntsville  (CEHND).  OE- 
KB  involves  two  levels  of  operation  :  Detection  and  Discrimination. 

15.1.1  Detection  Approach 

Detection  involves  locating  potential  anomalies.  This  is  typically  an  automatic  operation, 
using  algorithms  developed  specifically  for  this  purpose.  Existing  automatic  techniques 
were  found  to  be  ineffective  in  some  data  sets  owing  to  a  significant  noise  component. 
Using  such  an  automatic  method  would  have  resulted  in  degraded  results  on  some  data 
due,  in  part,  to  the  picking  algorithm.  Therefore,  anomaly  detection  was  conducted 
manually  at  JPG  HI  to  ensure  consistent  and  complete  results. 

To  aid  in  visually  separating  potential  anomalies  from  the  back  ground  noise  several 
filters  were  developed.  Two  types  of  "Gradient  Focusing"  were  used  along  with  a  slightly 
smoothed  raw  data  representation  to  identify  anomalies.  Figure  1  is  raw  EM-61  data 
from  JPG  in.  Figure  2  is  gradient  focused  data  at  the  same  scale:  the  improvement  in 
anomaly  detectability  is  dramatic  (figures  not  present  in  MS  Word  document  -  refer  to 
WordPerfect  document) 
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After  these  processing  steps,  anomalies  in  the  magnetic  data  were  picked  for  each 
scenario  then,  if  there  was  an  EM-61  data  set,  EM-61  anomalies  were  picked.  The  result 
is  a  CAD  file  with  white  target  cells  for  magnetic  anomalies  and  yellow  target  cells  for 
those  anomalies  not  seen  with  the  mag  but  seen  by  the  EM. 

15.1.2  Discrimination  Approach 

SC&A's  approach  to  discrimination  is  an  attempt  to  optimize  the  tradeoff  between 
reducing  excavation/project  cost  and  the  risk  of  leaving  ordnance  in  the  ground.  Based 
upon  prior  experience,  SC&A  has  adopted  some  basic  premises  to  accomplish  this 
optimization: 

1.  The  vast  majority  of  metallic  targets  at  OE  sites  are  smaller  than  the  definition  of 
ordnance  at  a  particular  site  (less  than  1  LB,  for  example).  The  number  of  holes  dug  can 
be  significantly  reduced  at  minimum  risk,  if  this  target  population  is  identified  and 
eliminated  from  excavation. 

2.  The  limited  number  of  large  targets  and  the  risk  associated  with  misidentification  of  a 
large  target  as  non-UXO  minimizes  the  cost/benefit  of  winnowing  the  target  population 
from  this  subset  of  the  target  population. 

3.  Noise  levels  in  EM  and  magnetometer  data  are  commonly  sufficient  to  degrade 
anomalies  produced  by  targets  near  the  low  mass-limit  of  interest  (those  outlined  in 
premise  1).  Algorithms  such  as  gradient  focusing,  neural  and  fuzzy  systems  etc.,  tailored 
to  site  specific  noise  conditions,  are  required  to  reliably  separate  targets  directly  above 
and  below  the  low  mass  limit. 

15.1.3  Discrimination  Implementation 

15.1.3.1  Phase  I 

The  process  of  picking  anomalies  on  multiple  data  sets,  described  above,  provides  a 
direct  discrimination  between  ferrous  and  nonferrous-metallic  anomalies.  This 
accomplishes  an  initial  subdivision  of  the  anomaly  population. 

15.1.3.2  Phase  II 

Slightly  smoothed  magnetic  and  EM-61  data  sets  are  displayed  side  by  side  on  a  screen 
and  each  anomaly  is  classified  as  either  ordnance,  non-ordnance,  or  unknown,  using  the 
conservative  approach  described  above. 

15.1.3.3  Phase  III 

Due  to  time  restrictions  this  level  of  the  discrimination  system  was  applied  only  to 
scenario  4  data.  An  adaptive  neural  fuzzy  system  is  used  to  estimate  mass,  based  on 
signal  returns  from  both  coils  of  the  EM-61. 
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15.1.3.4 


Phase  IV 


A  data  fusion  system  that  uses  data  from  both  the  EM-61  and  the  magnetometer  could  not 
be  applied:  unanticipated  corrections  of  data  from  the  participants  imposed  time 
limitations  that  could  not  be  overcome. 

15.1.4  Limitations 

During  previous  site  remediation  actions  SC&A  was  able  to  establish  precise  definition  of 
the  lower  mass  limit  through  an  iterative  update  of  the  database.  This  was  achieved  by 
associating  anomalies  with  known  targets  through  the  progression  of  site  excavations. 
Such  an  iterative  optimization  of  the  lower  mass-limit  could  not  be  achieved  at  JPGin  as 
no  excavations  were  done.  Hence,  the  inability  to  precisely  distinguish  masses  on  the 
lower  side  of  the  low  mass-limit  diminishes  SC&A's  ability  to  pare  false  alarms  from  the 
JPGin  data. 

15.1.5  Methodology 

In  an  effort  to  prevent  cross-contamination  of  results,  each  data  set  was  processed  and 
analyzed  independently.  The  order  for  processing  was  GeoCenters,  ADI  and  Geometries. 
The  results  for  each  participant  were  compiled  and  sent  to  PRC  before  the  next  data  set 
was  started.  After  each  data  set  was  sent  to  PRC,  that  participant’s  data  and  results  were 
removed  from  the  computers  and  stored  on  a  backup  device:  results  from  different 
companies  were  not  compared. 

To  ensure  that  each  data  set  was  evaluated  similarly,  a  single  analyst  evaluated  all  of  the 
data  sets.  Because  of  the  very  large  number  of  anomalies  evaluated  it  was  felt  that 
operator  bias  would  not  be  a  factor  in  anomaly  location:  this  proved  to  be  the  case. 

15.1.6  Performance  Differences 

SC&A  found  differences  in  data  quality  between  the  participants  for  given  types  or 
classes  of  instruments.  These  differences  impacted  the  amount  of  SC&A’s  effort  required 
to  produce  reliable  test  results.  This  level  of  effort  will  likely  not  be  apparent  in  the  JPG 
in  results. 

Different  companies  elected  different  instruments  or  instrument  suites  on  the  different 
scenarios  (Table  1).  SC&A  analyzed  the  data  set(s)  provided  by  each  company,  for  each 
scenario,  as  a  whole  to  obtain  the  best  results  possible.  Data  type  differences  yielded 
different  results.  For  example:  ADI  proposed  and  used  a  magnetometer  for  scenario  1 
and  the  results  for  ADI  in  scenario  1  reflect  this  survey  approach.  GeoCenters  proposed 
and  used  both  a  magnetometer  and  an  EM-61.  Results  for  GeoCenters  reflect  this 
combination  of  instruments.  Thus,  results  from  individual  scenarios  for  different 
companies  are  not  related  to  instrument  quality  or  survey  methodology  alone,  they  also 
reflect  the  instrument  choices  made. 
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Table  1.  Instrumentation  deployed  by  each  participant  for  each  scenario. 


Scenario  1  Scenario  2  Scenario  3  Scenario  4  | 

ADI 

MAG 

MAG 

MAG  and  EM-61 

MAG  and  EM- 
61 

GeoCenters 

MAG  and  EM-61 

MAG  and  EM-61 

MAG  and  EM-61 

None 

Geometries 

MAG  and  EM- 
61 
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15.1.7  Objective  3:  Demonstrate  effective  coordination  between  companies... 

Coordination  was  required  from  the  beginning  in  order  to  accomplish  this  unique 
proposal.  ADI,  GeoCenters  and  Geometries  agreed  to  participate  with  SC&A  in  the 
further  analysis  of  their  JPGin  data.  This  multicompany  coordination  is  reminiscent  of 
the  way  data  is  collected,  processed,  and  interpreted  in  the  oil  industry.  Indeed,  the  OE- 
KB  was  designed  with  this  type  of  service  segmentation  in  mind.  In  order  to  facilitate 
this  cooperative  model  SC&A  believes  that  adoption  of  a  near  surface  geophysical  data 
standard  is  necessary.  Such  a  standard  would  eliminate  time  consuming  effort  expended 
to  translate  a  particular  vendor's  data  Into  a  format  compatible  with  another  vendors  data, 
processing,  and/or  interpretation  requirements.  In  addition  to  facilitation  of  data  transfer, 
such  a  standard  would  also  enable  easy  reevaluation  (processing  or  otherwise)  of  old  data 
in  light  of  new  data,  results,  processing  methods,  and/or  interpretation  methods. 

Efforts  aimed  at  developing  specialized  expertise  and  standardization  are  leading  to  lower 
overall  costs  and  more  effective  operations.  SC&A  considers  this  "experiment"  to  have 
been  a  success  and  looks  forward  to  further  coordination  in  the  future. 

V 

15.1.8  Data  Processing  System 

15.1.8.1  Hardware 

Pentium  Class  Personal  Computers  with  minimum  of  64  Mb  Ram  were  used. 

15.1.8.2  Software 

The  following  software  was  used  for  various  phases  of  the  analysis; 

1 .  Intergraph  MGE  software  suite 

2.  OE-KB,  A  CEHND  product  for  analysis  of  OE  geophysical  anomalies 

3.  MATLAB 

4.  Various  programs  for  manipulating  ASCII  data  series  including  AWK  and  MS  Excel 

5.  Custom  programs  for  data  manipulation  and  processing 

6.  Personnel 


15.1.8.3  Personnel 

Matthew  Gifford  -  Project  manager  and  data  analyst 
Jack  Foley  -  Software  design  and  program  management 
Dave  Lieblich  -  Software  design  and  data  pre-processing 
Cynthia  Saine  -  Data  pre-processing 
Brian  Coolidge  -  Data  pre-processing 
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SCA_ADI  -  Combined  Statistics:  Scenarios  1,2,3 


Detection  Statistics  (TMA°  Group) 


#  Baseline 

#  Detected 

Ordnance 

208 

132 

0.63  0.07  1 

Nonordnance 

166 

110 

0.66 

Total 

374 

242 

Number  False  Alarms 

575 

False  Alarm  Rate  (#/Hectare) 

46.80 

False  Alarm  Ratio  (#/Ord.) 

4.36 

Probability  False  Alarms 

0.059 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation 

Position  (x,v) 

dx  -  eastina  error 

0.02 

0.45 

dv  -  northing  error 

-0.08 

0.46 

Radial  error 

0.54 

0.35 

Depth  (z) 

dz  -  averaged  depth  error 

-0.14 

0.34  I 

IdzP-  absolifte  depth  error 

0.36 

Characterization  Statistics  (TMA  Closest) 


#  Baseline 

#  Detected 

Po-’ 

#  Correct 

Pc“ 

Abilitv  to  Tvpe 

Ordnance 

208 

132 

0.63 

124 

0.94 

Nonordnance® 

167 

110 

0.66 

1 

0.01 

Large 

14 

14 

1.00 

2 

0.14 

Medium 

39 

32 

0.82 

14 

0.44 

Small 

155 

86 

0.55 

80 

0.93 

Abilitv  to  Classify  1 

Bomb 

21 

21 

1.00 

0 

0.00 

Projectile 

42 

38 

0.90 

0 

0.00 

Mortar 

26 

18 

0.69 

0 

0.00 

Submunition 

97 

44 

0.45 

43 

0.98 

Rocket 

22 

11 

0.50 

0 

0.00 

Notes:  “  Target  Matching  Algorithm 

^  Probability  of  detection 
Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
^MA  affects  how  nonordnance  baseline  targets  are  counted 
NA '  not  appiicable 


B.15-6 


Depth  (meters) 


JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Crfticai  Radius:  2  meters 
Demonstrator:  SCA.ADI 

Small  Medium  Large 


Principal  Diameter  (millimeters) 


■  Target  Detected 
a  Target  Not  Detected 


B.15-7 


SCA_Geo-Centers  -  Combined  Statistics:  Scenarios  1,2,3 


Detection  Statistics  (TMA*  Group) 


#  Baseline 

Ordnance 

208 

Nonordnance 

166 

112  0.67 

Total 

374 

271 

Number  False  Alarms 

535 

False  Alarm  Rate  (#/Hectare) 

43.55 

False  Alarm  Ratio  (#/Ord.) 

3.36 

Probability  False  Alarms 

0.055 

Localization  Statistics  (TMA  Closest)  (in  meters) 


Mean 

Std  Deviation 

dx  -  eastinq  error 

-0.27 

0.57 

dy  -  northing  error 

0.22 

0.52 

Radial  error 

0.74 

0.41 

Depth  (z) _ _J 

dz  -  averaged  depth  error 

-0.14 

0.34 

Idzl'-  absolute  depth  error 

0.36 

Characterization  Statistics  (TMA  Closest) 


1  #  Baseline  |  #  Detected  |  Po*’  |#  Correct  Pc'"  1 

Ordnance 

208 

159 

0.76 

147 

0.92 

Nonordnance® 

167 

112 

0.67 

3 

0.03 

lAbilitv  to  Size  J 

Large 

14 

14 

1.00 

5 

0.36 

Medium 

39 

36 

0.92 

13 

0.36 

Small 

155 

109 

0.70 

101 

0.93 

Abilltv  to  Classify  J 

Bomb 

21 

19 

0.90 

0 

0.00 

Projectile 

42 

37 

0.88 

0 

0.00 

Mortar 

26 

15 

0.58 

0 

0,00 

Submunition 

97 

67 

0.69 

67 

1.00 

Rocket 

22 

21 

0.95 

0 

0.00 

Notes:  *  Target  Matching  Algorithm 

Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
**  Probability  of  correct  characterization 
^MA  affects  how  nonordnance  baseline  targets  are  counted 
NA  -  not  applicable 


B.15-8 


JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  Ail  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator:  SCA^Geocenters 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 
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SCA_Geometrics  -  Combined  Statistics:  Scenario  2  only 


Detection  Statistics  (TMA*  Group) 


^ - 

#  Baseline 

#  Detected 

Pd*^  1  P  random 

Ordnance 

67 

64 

0.96  1  0.068 

Nonordnance 

50 

46 

0.92 

Total 

117 

110 

Number  False  Alarms 

196 

False  Alarm  Rate  (#/Hectare) 

41.86 

False  Alarm  Ratio  (#/Ord.) 

3.06 

Probability  False  Alarms 

0.053 

Localization  Statistics  (TMA  Closest)  (In  meters) 


_ 

Mean 

Std  Deviation 

dx  -  eastina  error 

-0.09 

0.35 

dv  -  northing  error 

0.11 

0.45 

Radial  error 

0.49 

0.32 

Death  (z) 

dz  -  averaged  depth  error 

-0.04 

0.26  1 

Idzl®-  absolute  depth  error 

0.26 

Characterization  Statistics  (TMA  Closest) 


#  Baseline  |  #  Detected  Pd"  |#  Correct  |  Pc”  I 

Abllitv  to  Type 

Ordnance 

67 

64 

0.96 

58 

0,91 

Nonordnance 

50 

46 

0.92 

3 

0.07 

1 1  nil  'll  1  II 1  ^—i 

Large 

3 

3 

1.00 

2 

Medium 

31 

29 

0.94 

18 

Small 

33 

32 

0.97 

18  1 

0.56 

lAbilitv  to  Classify 

Bomb 

0 

0 

NA 

0 

NA 

Projectile 

41 

39 

0.95 

0 

0.00 

Mortar 

26 

25 

0.96 

0 

0.00 

Submunition 

0 

0 

NA 

0 

NA 

Rocket 

0 

0 

NA 

0 

NA 

Notes:  ‘  Target  Matching  Algorithm 

Probability  of  detection 
°  Square  root  of  the  mean  square  depth  error 
Probability  of  correct  characterization 
NA  -  not  applicable 
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Depth  (meters) 


JPG  Controlled  Site:  Phase  III 

Combined  Statistics:  All  Surveyed  Area  (Excluding  Seen.  4) 
Critical  Radius:  2  meters 
Demonstrator:  SCA_Geom9trlcs 

Small  Medium  Large 


■  Target  Detected 
□  Target  Not  Detected 
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